Coded localization systems, methods and apparatus

ABSTRACT

A coded localization system includes a plurality of optical channels arranged to cooperatively image at least one object onto a plurality of detectors. Each of the channels includes a localization code that is different from any other localization code in other channels, to modify electromagnetic energy passing therethrough. Output digital images from the detectors are processable to determine sub-pixel localization of the object onto the detectors, such that a location of the object is determined more accurately than by detector geometry alone. Another coded localization system includes a plurality of optical channels arranged to cooperatively image partially polarized data onto a plurality of pixels. Each of the channels includes a polarization code that is different from any other polarization code in other channels to uniquely polarize electromagnetic energy passing therethrough. Output digital images from the detectors are processable, to determine a polarization pattern for a user of the system.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. provisional application Ser.No. 61/631,389, filed Jan. 3, 2012, entitled ANGULAR CODING FOR 3DOBJECT LOCALIZATION; U.S. provisional application Ser. No. 61/634,421,filed Feb. 29, 2012, entitled ANGULAR CODING FOR 3D OBJECT LOCALIZATION;U.S. provisional application Ser. No. 61/634,936, filed Mar. 8, 2012,entitled SYSTEMS AND METHODS FOR MOTILITY AND MOTION OBSERVATION ANDDISCRIMINATION; U.S. provisional application Ser. No. 61/685,866, filedMar. 23, 2012, entitled ANGULAR CODING FOR 3D OBJECT LOCALIZATION; U.S.provisional application Ser. No. 61/686,728, filed Apr. 11, 2012,entitled ANGULAR CODING FOR 3D OBJECT LOCALIZATION; U.S. provisionalapplication Ser. No. 61/687,885, filed May 3, 2012, entitled ANGULARCODING FOR 3D OBJECT LOCALIZATION; U.S. provisional application Ser. No.61/655,740, filed Jun. 5, 2012, entitled ANGULAR CODING FOR 3D OBJECTLOCALIZATION AND MOTION DETECTION; U.S. provisional application Ser. No.61/673,098, filed Jul. 18, 2012, entitled ANGULAR CODING FOR 3D OBJECTLOCALIZATION; U.S. provisional application Ser. No. 61/692,540, filedAug. 23, 2012, entitled ANGULAR CODING FOR 3D OBJECT LOCALIZATION; U.S.provisional application Ser. No. 61/720,550, filed Oct. 31, 2012,entitled POLARIZATION CODING; and U.S. provisional application Ser. No.61/729,045, filed Nov. 21, 2012, entitled POLARIZATION CODING. All ofthe above-identified patent applications are incorporated by referenceherein, in their entireties.

U.S. GOVERNMENT RIGHTS

This invention was made with U.S. Government support under contractW911NF-11-C-0210 awarded by the U.S. Army, UCSD PO #10320848. TheGovernment has certain rights in this invention.

BACKGROUND

Estimating and detecting the 3D location and orientation of objects is aclassic problem. Numerous prior art systems have issues which negativelyinfluence the achievable size, weight, and power, cost and precision ofobject 3D localization and orientation.

A fundamental issue with prior art optical digital imaging systems shownin FIG. 1 (upper drawing) that perform 3D localization is thatresolution is related to the spatial density of image sensing pixels:specifically, pixels with fixed system geometries limit higherprecision. In general prior art digital imaging systems, when the lensis at its maximum diameter, the image size formed behind the lens, forparticular sized pixels, is a maximum; and put another way, thespace-bandwidth product (SBP) at the image plane is a maximum. Withsmaller diameter lenses, but with a constant marginal ray angles or F/#,smaller images are captured and the system's SBP is reduced. But as thelens diameter decreases, the image size becomes so small that forgeneral scenes, all object information is not captured in sufficientdetail.

Similarly in prior art measurement systems as shown in FIG. 1 (upperdrawing), the baseline B between sensors determines the SPB, and henceprecision of estimating x, y, z values for object 105 in system 100.Optics 110 a and 110 b, with focal lengths fa and fb, respectively, formimages of object 105 at distances ya and yb, respectively, from theoptical axis on their respective sensors 120 a and 120 b. The estimatesof distances ya and yb determine y location and ratios of fa/ya andfb/yb determine the range, thus a more dense sampling at sensors 120 aand 120 b increases the precision of the estimations—but at the cost ofhigher size, weight and power.

Like the two elements of a stereo imaging system shown in the upperdrawing of FIG. 1, prior art radar systems (see lower drawing of FIG. 1)also use arrays of detectors to detect objects in 3D space. Radarsystems code across each element of a multi-element antenna array inorder to detect/localize/reject, etc. targets in a 3D space in thepresence of unknown noise and clutter. The number of antenna elementsdoes not theoretically limit the potential angular estimation accuracy,except through SNR.

The main differences between the radar system (lower drawing) of FIG. 1and the optics system (upper drawing) of FIG. 1 is related to the factthat typical radar wavelengths can be temporally coherently sampled withantenna 102 a, 102 b, to 102 n. In radar systems the antenna elementsare arranged so that the field patterns from each element overlap.Mathematical weights 104 a, 104 b, to 104 n are applied across the arrayelements or channels and then summed 106 to determine the output signalsquared 107. The weights can be deterministic, such as Fouriercoefficients, or can be a function of the sampled data, such as used inAuto-Regressive (AR) modeling, and can have a representation on thecomplex unit circle 108 in FIG. 1 where both signal amplitude and objectangle can be represented.

SUMMARY OF THE INVENTION

The limitations of prior art localization imaging systems are improvedas disclosed herein, where an optical array of elements is arranged moresimilarly to a radar system, such that the field patterns from eachelement or channel overlap. Mathematical weights may also be appliedacross the array, through embodiments of coded localization optics orspecialized coded localization sensors described hereinbelow. Decodingprovides the information desired, enabling optical imaging systems toperform localization tasks in a manner superior in performance, size,weight, and power to traditional incoherent stereo-like imaging systems(FIG. 1). In contrast to the problems identified in the prior art above,with coded localization systems disclosed hereinbelow, spatial andangular resolution are determinable by signal-to-noise ratio and not bythe spatial resolution of actual image pixels. Uncoupling resolution andpixel density or sampling provides for optical systems that reduce size,weight cost and power and yet perform localization, orientation, andimage formation.

In an embodiment, coded localization systems may include optics, codes,sensors and processors where the localization precision is related tothe joint design of at least two of optics, codes, and sensors.

In an embodiment, coded localization systems form N samples of a remoteobject with each sample being coded with a unique code.

In an embodiment, coded localization systems are designed so that thecross-information between channels sampling the same remote object isreduced compared to no coding.

In an embodiment, the channels consist of at least one of an image,aperture, wavelength, polarization, and field of view.

In an embodiment, coded localization systems where the codes in multiplemeasurements employ sinusoidal functions.

In an embodiment, coded localization systems where the unbiased codessum to zero.

In an embodiment, coded localization systems where the N measurementsform a measurement vector as a function of spatial position.

In an embodiment, coded localization systems where the N measurementsform a measurement vector with a magnitude and phase as a function ofspatial position.

In an embodiment, coded localization systems where the unbiased codesform an orthogonal set.

In an embodiment, coded localization systems where the sampled PSF ofthe imaging system is space-variant.

In an embodiment, coded localization systems the measured samples varyas a function of sub-pixel changes in the aerial image.

In an embodiment, coded localization systems where the measurementvector is compared to a stored or estimated object vector to detect theobject and orientation and/or location.

In an embodiment, coded localization systems where the measurementvector is used with a mathematical model of the object to estimateobject parameters.

In an embodiment, coded localization systems where measurements are madein at least one of spatial, temporal, spectral and/or polarizationdomains.

In an embodiment, coded localization systems where measurements are madein at least two of spatial, temporal, spectral and/or polarizationdomains.

In an embodiment, coded localization systems where multiple measurementsof a remote object are made simultaneously in at least one domain oftemporal, spectral, spatial and polarization.

In an embodiment, coded localization systems where multiple measurementsof a remote object are made simultaneously in at least two domains oftemporal, spectral, spatial and polarization.

In an embodiment, coded localization systems where measurements are madein the visible wavelengths.

In an embodiment, coded localization systems where measurements are madein IR wavelengths.

In an embodiment, coded localization systems where the N measurementsare configured to be unbiased.

In an embodiment, coded localization systems where the N measurementsare designed so that the sum of the squares of the N measurements isconstant in the absence of noise.

In an embodiment, coded localization systems where the uncertainty inspatial localization is less than the area of a detector pixel.

In an embodiment, coded localization systems where the optics reduce thememory storage required compared to the FOV and localizationuncertainty.

In an embodiment, coded localization systems where optics anddata-dependent signal processing reduces the amount of data storedcompared to classical imaging systems.

In an embodiment, coded localization systems where multiple systems areused for at least one of orientation and 3D localization.

In an embodiment, coded localization systems where multiple systems areused to estimate change in at least one of orientation and 3D location.

In an embodiment, coded localization systems where the estimated ordetected change in orientation or location is finer than the pixelspacing.

In an embodiment, coded localization systems where complimentarymeasurement vectors are used to estimate change in at least one oforientation and 3D location.

In an embodiment, coded localization systems provide measurement vectorsthat represent Fourier components beyond the spatial frequency limits ofthe detector pixels.

In an embodiment, coded localization systems where single pixel sensorsand multiple measurements are formed through spatially displacedsystems.

In an embodiment, coded localization systems where single pixel sensorsand multiple measurements are formed through temporally displacedsystems

In an embodiment, coded localization systems where the coding is bothspatial and temporal.

In an embodiment, coded localization systems where the complimentarychannels are chosen in order to reduce the length of the overall opticalsystem.

In an embodiment, coded localization systems provide complimentarymeasurements which are combined in digital processing to form a finalimage with resolution corresponding to the total number of measurements.

In an embodiment, a method and apparatus are provided for performinglong range photo point measurements, with high precision compared todetector pixel spacing.

In an embodiment, a method and apparatus are provided forcontext-dependent optical data reduction and storage.

In an embodiment, a method and apparatus are provided for relativeobject location estimation with precision high compared to the number ofpixels.

In an embodiment, a method and apparatus are provided for absoluteorientation estimation.

In an embodiment, a method and apparatus are provided for absolute 3Dlocation estimation.

In an embodiment, a method and apparatus are provided to transfer atleast one of location and orientation reference information from a knownpoint to a new point.

In an embodiment, a method and apparatus are provided to for datacollection and data reduction for highway information.

In an embodiment, a method and apparatus are provided to for orientationand 3D location estimation for machine control.

In an embodiment, an improvement is provided for orientation and 3Dlocation estimation for positioning.

In an embodiment, a method and apparatus for performing at least one oforientation and 3D location estimation for indoor applications.

In an embodiment, a method and apparatus for performing at least one oforientation and 3D location estimation for applications with unreliableGPS coverage.

In an embodiment, a method and apparatus for performing at least one ofdistant object orientation and 3D location estimation for input to modelbuilding process.

In an embodiment for a two pixel sensor the measures of angle to theobject are used to count the number of objects in the field of view.

In an embodiment for a two pixel sensor the measures of angle frommultiple sensors are used to count the number of objects in the combinedfield of view.

In an embodiment by coding each channel differently more than one priorart pixel of information can be captured.

In an embodiment by coding each channel as a function of range afocusing can be achieved for objects not at infinity.

In an embodiment by coding each channel as a function of range anextended depth of field image can be achieved.

In an embodiment by coding each channel as a function of range anestimate of range for points in the image can be achieved.

In an embodiment an optical motion unit provides dead reckoning fornavigation.

In an embodiment, a coded localization system includes a plurality ofoptical channels arranged to cooperatively image at least one objectonto a plurality of detectors. Each of the optical channels includes alocalization code to optically modify electromagnetic energy passingtherethrough, and each localization code is different from any otherlocalization code in other optical channels. Output digital images fromthe detectors are processable to determine sub-pixel localization ofsaid object onto said detectors, and such that location of the object isdetermined more accurately than by detector geometry alone.

In an embodiment, a coded localization system includes a plurality ofoptical channels arranged to cooperatively image partially polarizeddata onto a plurality of pixels. Each of the optical channels includes apolarization code to uniquely polarize electromagnetic energy passingtherethrough, each polarization code being different from any otherpolarization code in other optical channels. Output digital images fromthe detectors are processable, to determine polarization pattern for auser of the system.

In an embodiment, a coded localization system includes a plurality ofoptical channels arranged to cooperatively image a moving scene onto aplurality of pixels. Each of the optical channels uniquely determinesits two dimensional change in motion of the scene. A rigid body modelcouples output from the channels to constrain a global change inlocation and orientation to a physical motion. A processing subsystemdecomposes data from each channel into a global motion vector.

In an embodiment, a coded localization system includes a plurality ofcoded localization channels wherein the system has Fisher Informationgreater than an optical system without localization codes.

In an embodiment, a method of localizing optical data includescooperatively imaging at least one object onto a plurality of detectorswhile implementing localization codes uniquely to each optical channel,and processing output digital images from the detectors to determinesub-pixel localization of said object onto said detectors such thatlocation of the object is determined more accurately than by detectorgeometry alone.

BRIEF DESCRIPTION OF DRAWINGS

The upper drawing in FIG. 1 is an example of a prior art opticallocation estimation system; while the lower drawing of FIG. 1 is anexample radar based prior art localization system.

The upper drawing of FIG. 2 shows an embodiment of one codedlocalization system for location estimation; while the lower drawing ofFIG. 2 provides a radar analogy.

FIG. 3 shows an embodiment of a coded localization system, and a sensingand processing method in accord with another embodiment.

FIG. 4 shows an embodiment of a coded localization system, and a sensingand processing method with an unknown medium with another embodiment.

FIG. 5 shows an embodiment of one two pixel bipolar coded localizationsystem.

FIG. 6 shows an embodiment of one coded localization system for imagingon a detector with overlapping fields of view.

FIG. 7 is an embodiment of a coded localization system using singlepixel detectors.

FIG. 8 is an embodiment of a coded localization system using a prioriinformation.

FIG. 9 is an embodiment of a coded localization system for motility andmotion observation and discrimination.

FIG. 10 shows graphs of a 3 channel linear combination codedlocalization system that increases the Fisher Information on objectlocation.

FIG. 11 shows graphs of codes for a coded localization system performingsub-pixel imaging of a 1D line on a set of coded pixels.

FIG. 12 illustrates codes for a phased sinusoidal coded localizationsystem responding to a sub-pixel edge.

FIG. 13 is an embodiment of a coded localization motility and motionobservation and discrimination system with overlapping fields of view.

FIG. 14 is an embodiment of a coded localization system showing 8polarization steps with 2pi symmetry, or pi/4 phase steps, or pi/8 realrotation steps of a linear polarizer.

FIG. 15 is an embodiment of the invention including stacking samples andperforming spectral decomposition.

FIG. 16 is an embodiment of the invention including stacking samples andperforming spectral decomposition.

FIG. 17 is an embodiment of the invention where the amplitude and biasare optimized for increasing exposure.

FIG. 18 is an embodiment of a localization code that can be appliedacross the field of view of a bipolar two detector motion sensor system.

FIG. 19 illustrates an embodiment where at least two coded localizationsystems can be used to estimate object range and velocity in addition toobject angle and angular velocity from object motion.

FIG. 20 is an embodiment of a coded localization system for motility andmotion observation and discrimination.

FIG. 21 illustrates an embodiment of the invention for motility andmotion observation and discrimination.

FIG. 22 is an embodiment of the invention with sampling diversity forcoded localization systems.

FIG. 23 illustrates an embodiment of coded localization systems formingimages using overlapping FOV 2×2 systems.

FIG. 24 is an embodiment of a design method for generalized codedlocalization systems.

FIG. 25 is an embodiment of an optimized two pixel system.

FIG. 26 is an embodiment of the invention and describes the constructionof a code that enables both narrow FOV and wide FOV localization.

FIG. 27 is one embodiment of a 4 channel linear combination system thatincreases the Fisher Information on object location.

FIG. 28 is an embodiment of the invention that describes edge directionestimation.

FIG. 29 is an embodiment of the invention for optimizing the designspace for weight and number of pixels.

FIG. 30 is an embodiment of a design method and trade space.

FIG. 31 is an embodiment of the invention disclosing a measurementapplication.

FIG. 32 is an embodiment of the invention disclosing a measurementapplication.

FIG. 33 is an embodiment of the invention disclosing a measurementapplication.

FIG. 34 is an embodiment of the invention disclosing measuring andcontrol using coded localization systems.

FIG. 35 is an embodiment of the invention disclosing measuring andrecording using coded localization systems.

FIG. 36 shows the use of coded imaging channels to add information forhuman viewers.

FIG. 37 shows the mixing of time, space and polarization in order tofurther detection, recognition or estimation of signals.

FIG. 38 depicts a high resolution imaging system coupled to a codedlocalization system.

FIG. 39 shows an embodiment of a device for a polarization sky compassutilizing the coded designs.

FIG. 40 shows another embodiment of a device for a polarization skycompass utilizing the coded designs.

FIG. 41 describes a high-speed localization system that has milli-radianestimation precision, small size and low cost.

FIG. 42 shows a preferred embodiment of a device for imaging using thecoded designs and processing systems.

FIG. 43 shows a preferred embodiment of a device for imaging using thecoded designs and processing systems.

FIG. 44 is an embodiment of the invention disclosing ranging usinggroups of single pixels.

FIG. 45 illustrates focusing and codes as a function of range.

FIG. 46 is an embodiment of a navigation system for dead reckoning.

FIG. 47 illustrates a polarization compass, in an embodiment.

FIG. 48 illustrates an optical motion and motility unit, in anembodiment.

FIG. 49 illustrates details of the imaging systems of FIGS. 39-43.

FIG. 50 illustrates 2×2 and 3×3 systems, using a single sensor as adetector element for all channels, in an embodiment.

DETAILED DESCRIPTION OF EMBODIMENTS

The following disclosure is organized to first present general themes,and then design methods, and finally specific applications and devices.System concepts and mathematics underlying the various embodiments,including the physical layout of optical coding and data vectorformation are shown in FIGS. 2-4. Examples for applying codes tomultiple-pixel and single-pixel system are shown in FIGS. 5-9. Aframework for designing particular codes based on Fisher information isdisclosed in FIGS. 10-12, and provides methodology for selecting aparticular code for a task requirement. With the full reading andappreciation of FIGS. 2-12, specific devices may be developed. Suchdevices can embody several system embodiment features at the same time,and examples are provided. An example of data processing where the codesare polarization analyzer angles is shown in FIGS. 13-17. A two-pixeldetector example that detects the location of a moving object isprovided in FIGS. 18-20; this approach is extended to multi-aperturemulti-pixel systems in FIGS. 20-22 to detect motion and orientation. Thecoded systems disclosed hereinbelow may also form full-resolution imagesas initially disclosed in FIG. 23 and shown again in later applicationFIGS. 24-30. More complex codes and tasks with more apertures, withmethods of design which include cross-talk and SNR considerations, arepresented in FIGS. 24-30. Applications of enabled devices performingprofile measurements and localization tasks are provided in FIGS. 31-41.More details about the devices that may be used in these applicationsare included in FIGS. 42-50, along with example design parametersincluding codes, task requirements, processing algorithms, and numericalresults.

As noted above, coded localization systems are disclosed hereinbelow.Such imaging systems are useful for measuring relative and absolutelocation and orientation information as well as for image formation withreduced system length. The coding applied to form the coded localizationsystems may be task-dependent and optimized for information captureaccording to the detection or estimation task at hand. Codedlocalization systems and algorithms for passive infrared detection andpolarization detection and estimation are also disclosed herein.Techniques to reduce size, weight, and power in coded localizationimaging and measurement systems are also disclosed.

In this disclosure, a “code” or “coding” can be either a property ofelectromagnetic energy that emanates from one or more objects, or adevice that detects such property on a localized level. A “codedlocalization system” is therefore a system that utilizes multiple imagesthat are acquired by detectors sensitive to the “codes” and can imputecode-specific information to each of the images. The code-specificinformation can, in turn, be utilized to extract information (such as,for example, location of one or more objects within the scene) from themultiple images, that would not be available without the coding.

In certain imaging domains, the amount of relevant object informationcan become very small or sparse compared to the system SBP. In long waveinfrared imaging systems, only objects with certain temperatures canform information on the sensor. In some cases only a small number ofwarm objects will be in a scene. In another example, only objects thatare a certain color, for example signs on roadways, may be in a sceneand may be the desired information to be captured and estimated. Witheither optical or digital filtering by color the average scene canbecome sparse. In another example, the edges of objects may be theinformation of interest. In many man-made scenes the quantity andorientation of object edges can be considered sparse. Polarization andtemporally short domains can be considered sparse as well. Some systemsestimating and detecting the 3D location and orientation of objectsinvolve coding and decoding of signals in the polarization domain. Withcoded localization systems disclosed herein, specialized sensing,decoding and processing may be used within spatial, spectrum andpolarization domains to increase the ability to localize distant objectsor features, enabling robust answers to questions “where is that” or“where am I”.

When a particular class of object scenes can be considered sparse, lessthan the maximum number of image pixels is needed to accurately estimatethe location of objects. Said another way, a system with a reduced SBPcan be used to accurately estimate the location of the objects. Using asmaller sensor and coded localization also results in a smaller systemdimensionally. When 3D estimation precision, or information capturecapacity of the system, is not a function of the density of pixels, thenexploiting the sparsity of the scene in some domain directly leads tosmaller sensors, less power and less costly overall systems. For examplethe SWaP (size, weight and power) is reduced. In some cases theprecision of 3D location estimation can be higher with localizationcoding than without even when large numbers of pixels are used in thesystem that does not employ localization coding.

The upper drawing of FIG. 2 is an embodiment of the invention shown assystem 200. In this embodiment, optics 210 a and 210 b, with focallengths fa and fb, respectively, form coded images of object 205 atdistances ya and yb, respectively, from the optical axis on respectivesensors 120 a and 120 b. The estimates of distances ya and yb onlypartially determine y location, and ratios of fa/ya and fb/yb onlypartially determine range. In this embodiment the channels are codedwith codes 230 a and 240 a, and 230 b and 240 b and the coded imagesformed at sensors 220 a and 220 b are decoded to further determine ylocation and the range or z location of object 205. Such codes are forexample binary or real-valued intensity or phase screens, masks, andapertures, where the code varies as a function of spatial location inthe image or aperture. Codes may also include polarization sensitive andselective materials. In one illustrative embodiment, codes are placed inlight path locations between the object and the sensor, for examplebetween the object and the objective lens, between the aperture and thesensor, or distributed throughout the system.

The goals of coded localization systems include coding of specializedoptical systems across more than one aperture in order to optimizedetection and localization estimation, increasing the informationcapacity of the system, and reducing system size. Such systems alsobecome insensitive to knowledge of 3D system geometry and insensitive toenvironmental system tolerances. These systems enable spatially coherent3D estimation even when the systems are geometrically and dynamicallyvarying. These systems may include optics, codes, sensors and processorswhere the localization precision is related to the joint design of atleast two of optics, codes, and sensors. Localization precision isdependent on system SNR and not on the number of pixels.

A radar system analogy can be redrawn as an optical system by changingthe location of the complex weights, squaring operator, and processing,and is shown in the lower drawing of FIG. 2. The main differencesbetween radar (lower drawing of FIG. 2) and optics (upper drawing ofFIG. 2) relates to the fact that typical radar wavelengths can betemporally coherently sampled; yet optical wavelengths cannot yet betemporally coherently sampled. Wavelengths are spatially coherentlysampled by detectors 207 a, 207 b to 207 n. And in radar systems theantenna elements are arranged so that the field patterns from eachelement overlap. In accord with this disclosure, an optical array ofelements can be similarly arranged, such that the field patterns fromeach element or channel overlap. The mathematical weights 202 a, 202 b,to 202 n can be applied across the array, through various codedlocalization optics or specialized coded localization sensors. Thesensors 207 a to 207 n are temporally incoherent samplers and form themagnitude squared estimates of the electromagnetic energy. These samplesare then decoded with process 206 to estimate or detect parametersrelated to distant objects. The mathematical weights 202 a, 202 b, to202 n may also be deterministic, such as Fourier coefficients, or be afunction of the sampled data, such as used in Auto-Regressive (AR)modeling, and can have a representation on the complex unit circle 208where signal amplitude and object angle are displayed. An indirectbenefit of localization coding is that each imaging channel is dependenton other imaging channels for sub-pixel localization. In a prior-artclassical stereo imaging system (see FIG. 1), each imaging channel isindependent of other channels. This independence drives a 3Dlocalization error due to 3D system position uncertainties as well asuncertainties in object/image correspondence in the resulting images.

In certain embodiments of coded localization systems disclosed herein,remote systems are used together to increase location precision. Thegeometry of such systems may be static or dynamic, as when the systemsmove relative to each other. The 3D system geometry also does not needto be known to sub-pixel tolerances, unlike prior art stereo imagingsystems. From the nature of coded localization systems, the location ofthe systems is determinable by the data generated through imaging of thesame distant object.

In coded localization systems disclosed herein, a distant object isimaged by multiple imaging channels. In one embodiment, the imagingchannels may have different chief ray angles, or boresight angles, tothe object. In another embodiment, the fields of view of differentimaging channels overlap, at least in the volume where the distantobject is located. In contrast to prior art stereo imaging systems, eachimaging channel is not a classical imaging system outside of thegeometrical information just described; rather, each channel acts tocode the relative angular location of distant objects as measured at thesensor. The coded localization systems thus form N samples of a remoteobject with each sample being coded with a unique code. The differentcoded information, as measured at the image sensors, is combined toenable high precision, fast computation, and the ability to enable imagecorrespondence between the different image channels.

In an embodiment, coded localization systems are used to code angularinformation, relative to two or more sub-systems and/or two or moreimaging channels on the same sensor. The relative angle and distance toa distant object is decoded with incomplete knowledge of optics andsensors (through environmental tolerances) or sub-systems (for exampleif sub-systems are moving relative to each other). In anotherembodiment, coded localization systems are used to improve estimates ofangular location and distance in situations where knowledge of opticsand sensors are known. In yet another embodiment, coded localizationsystems are used to reduce the search space when estimating imagecorrespondence by constraining the search to regions of correspondinglocalization codes.

FIG. 3 shows an embodiment of a coded localization system 300. In FIG.3, coded localization system 300 is configured to record N channels withoptics 310 to 310 n and sensors 320 to 320 n. In this configuration,each channel records 2D spatial, and possibly color and temporalinformation, for a particular code 340 to 340 n. In one embodiment,channel codes 340 to 340 n differ at least by a different polarizationvalue or angle of polarization. In another embodiment, channel codes 340to 340 n differ at least by an intensity of spatially varyingattenuation. In another embodiment, channel codes 340 to 340 n differ atleast by a phase of spatially varying intensity. In another embodiment,channel codes 340 to 340 n differ at least by a phase of spatiallyvarying sinusoidal intensity. In another embodiment, channel codes 340to 340 n differ at least by a value of spatially varying optical pathlength. In another embodiment, the channels consist of at least one ofan image, aperture, wavelength, polarization, and field of view. Lens395 is an optional objective lens that cooperates with optics 310 to 310n to form a re-imaging configuration wherein the optics 310 to 310 nfunction as a micro lenses array to re-image portions of the fieldthrough codes 340 to 340 n. If lens 395 is not present, optics 310 to310 n form individual objective lenses for each of the coded imagingchannels.

In an embodiment, the data set 370 from system 300 is a threedimensional matrix M(x,y,p) where (x,y) are spatial variables 350 and pis at least one of a polarization, intensity, and optical path variable360. There are N planes Mi in the data set 370; that is, for i=1 to n,Mi=M1 to Mn. Color and temporal variables may be embedded into eachspatial domain matrix Mi through such configurations as Bayer colorfilter arrays, temporal averaging, etc. Temporal information may also bemade explicit through an additional dimension of the matrix M. In anembodiment, processing system 380 combines data 370 containing matrix Maccording to the signal to be extracted from the coding sequence 340 to340 n. In one embodiment, processing system 380 forms a sub-signals_(ij) using the dimension p as the first iterator, wheres_(ij)=[M(1,1,1) M(1,1,2) . . . M(1,1,n)], s₁₂=[M(1,2,1) M(1,2,2) . . .M(1,2,n)], . . . , s_(ij)=[M(i,j,1) M(i,j,2) . . . M(i,j,n)]. In oneembodiment, the final signal s_(tot) to be processed is formed byconcatenating the sub-signals, s_(tot)=[s₁₁ s₁₂ s₁₃ . . . s_(ij)] andanalyzed for spectral content. The spectral analysis is demonstrated insubsequent paragraphs to enable a polarization compass among otherdevices. In an embodiment, the spectral analysis may be at least one oftemporal and spatially varying and may include at least one of aspectrogram analysis, a space-frequency analysis, a scale-spaceanalysis, and a time-frequency analysis. In another embodiment, thesub-signals are averaged and the average is analyzed for spectralcontent. In another embodiment, the data M1 to Mn are processed to forma weighted sum of images M_(tot)=Σw_(i)M_(i) where i=1 n, and theweights w₁ to w_(n) are based on the codes 340 to 340 n. In anotherembodiment, the codes in multiple measurements employ sinusoidalfunctions.

FIG. 4 shows an embodiment of one coded localization system 400employing polarization codes. A number of illumination objects, 491 a,491 b, 491 c are modified by unknown source codes 492 a, 492 b, 492 cand form signals 493 a, 493 b and 493 c. In one embodiment, source codes492 a, 492 b, 492 c are for example linear polarizers. In oneembodiment, the number of these objects, source codes and signals isone, for example as may be modeled while viewing a very narrow portionof the polarized sky. In another embodiment, the number of theseobjects, source codes and signals is larger than one, for example whenviewing a wide field of view of the polarized sky the polarization maybe modeled by a very large number of objects, source codes and signalsthat vary in angle and magnitude across the sky. In one embodiment, thesignals 493 a, 493 b and 493 c emanating from the sources/source codesare describable in a spatial, temporal and spectral domain. A generallyunknown medium 480 between the sources/source codes and the sensingsystems 405 acts to corrupt the signals 493 a, 493 b and 493 c. In anembodiment, a specialized sensing system is a collection of opticallycoded systems 405 to 405 n containing codes 440 to 440 n, optics 410 to410 n, and detectors 420 to 420 n. In one embodiment, codes 440 to 440 nare implemented in the polarization domain through polarizationanalyzers. In another embodiment, codes 440 to 440 n are implementedthrough linear polarization with varying physical angles. In anotherembodiment, the collection of optically coded systems 405 to 405 n isdesigned to record the spectral, spatial, temporal and polarizationinformation such that accurate and reliable information is captured inthe presence of the unknown medium 480. In one embodiment, processingthe recorded data by process 470 is used to extract the neededlocalization information from the recorded data. In another embodiment,codes 440 to 440 n are implemented through amplitude masks above thesensor planes of detectors 420 to 420 n where the collection ofoptically coded systems 405 to 405 n is designed to record the spectral,spatial, and temporal information such that accurate and reliableinformation is captured in the presence of the unknown medium 480.

FIG. 5 shows an embodiment of one coded localization system 500. Lenssystem 510 contains several different fields of view L1, R1, L2, R2, L3,and R3. Detectors 520 and 521 outputs are combined in a bipolar fashionto produce signal 515 when system 500 is exposed to motion 505 in thesensing field. Processing system 570 processes information in signal515. In one embodiment, at least one of complimentary lens regions 550,551 and 552 is purposely designed so that the measured signals at theoutput of the dual-element sensor 520, 521 are different for the leftand right fields of view for one or more imaging channels. For example,region 550 may be partially attenuating, by screening or blocking atleast a portion of the outermost lens surface. Alternatively, region 551or 552 may be at least partially attenuating, so that the response ofdetector 521 or 520 respectively is altered.

Lens configuration 580 is an embodiment of an exemplary system with 6lenses labeled 1 through 6 covering a left and right half of a field ofview, where the halves are separated by dotted line 582 and 592 in graph590. Configuration 580 illustrates the front surface of lens system 510,as viewed from the sensing field. Configuration 580 illustrates a smallfield of view 584 for each lens segment, indicating that each segmenthave non-overlapping sensing regions. Per system 500, each of thenon-overlapping field regions is imaged to be overlapping on thedetector pair 520 and 521. An attenuating mask is placed over lens 3.Example output 590 indicates a sinusoidal-like response for each of theL and R (or + and −) detector pixels within regions labeled 1 through 6in graph 590 corresponding to each of the lenses labeled 1 through 6 in580. Signal 595 is clearly attenuated peak-to-peak within region 3,indicating that the moving object was in the zone of lenses 1-3 coveringa half of the field of view. In this case the difference in the signalamplitudes indicates the angular location or field location of themoving object. Without the coded lens (i.e., in the prior art), allfield points will appear identical and angular location cannot bedetermined. In one embodiment, the code may be enabled through differentaperture areas and shapes for the left and right fields of view L3 andR3 for one or more imaging channels. In another embodiment, thedifference is obtained using at least one of different absorptive lensthicknesses, different polarizations, and polarizations purposelymatched or unmatched to detector element polarizations. In oneembodiment, detectors 520, 521 are thermally sensitive elements andsystem 500 forms a passive infrared motion sensing system. In anotherembodiment, lens system 510 is a Fresnel lens which can readily beformed inexpensively with optical molding and is highly compatible withdifferent absorptive lens thicknesses, different polarizations, andpolarizations purposely matched or unmatched to detector elementpolarizations. In one embodiment, the processing system 570 processessignal 515 to detect the signature caused by the localization coding inregion 550 to produce a measure of angle to the object that caused themotion. In an embodiment, for a two pixel sensor 520, 521, the measuresof angle to the object are used to count the number of objects in thefield of view. In an embodiment, for a two pixel sensor 520, 521, themeasures of angle from multiple sensors are used to count the number ofobjects in the combined field of view.

FIG. 6 shows an illustration 600 with an embodiment of codedlocalization system 604, along with expanded detail inside system 604.Apparatus 604 is a 2×2 coded localization system for imaging on detector620 with overlapping fields of view 602, the “2×2” notation referringherein to layout of imaging channels in a coded localization system.Each aperture in lens 610 is coded by a mask 641, 642, 643, 644 placedbetween lens 610 and sensor 620. In one embodiment, the masks aredifferent for each channel relative to the orientation of system 604. Byimaging the same object points multiple times with the example 2×2configuration, each object point is imaged 4 times. By coding eachchannel differently, more information can be captured (as compared to aprior art pixel architectures). In an embodiment, the aerial imagesub-pixel instantaneous fields of view from a particular object point,for this 2×2 example, may be given by I_(s)=[i₁₁ i₁₂; i₂₁ i₂₂]. Theaerial image sub-pixel instantaneous fields of view I_(s) are weightedby a mask and summed together on sensor 620 to form the coded sample673. In one embodiment, all four fields of view 602 may be formed asimage circles 621 on a single detector array 620. Each of the fourchannels has essentially the same information at the propercorresponding x/y locations before coding due to the overlapping fieldsof view 602. In one embodiment, the localization codes on masks 641,642, 643, 644 for the different imaging channels have differentfunctions and include at least one of spatial phase, spatial amplitude,polarization and optical path difference. In another embodiment, themask locations vary in the distance they are located between the lens610 and the sensor 620 (e.g., the mask for any one channel is located ata different location along the optical path as compared to the mask inanother channel). In another embodiment, the codes are composed of atleast one of a smooth function, a rapidly varying function, and randomor pseudo random functions that provide a broad set of basis functionsfor designing codes with many degrees of freedom. In another embodiment,the codes for a 2×2 implementation is a code composed of biasedsinusoids such that the code maximizes the Fisher Information of ageneral scene. In another embodiment, the measured samples vary as afunction of sub-pixel changes in an aerial image, enabling sub-pixelresolution measurements, or measurements at a finer scale than the pixelspacing. In another embodiment, the codes include wavelength specificcodes. Masks 641, 642, 643, 644 may select wavelengths, in oneembodiment, and be sensitive to a selection of wavelengths in anotherembodiment. By selecting a wavelength the codes can performhyper-spectral filtering as well as sub-pixel location information. Bybeing sensitive to a selection of wavelengths the codes can be used inpolychromatic imaging systems such as visible spectrum (red/green/blue)and affect all colors equally, thus preserving color fidelity, or affectcolors differently, providing another degree of coded informationthrough color variation while still being a broad spectrum device. In anembodiment, each object point is imaged in at least two of a selectionof visible wavelengths, near-IR, mid-wave IR, and long-wave IRwavelengths for broad spectrum imaging and use in imaging and targetingsystems that fuse at least two of visible, near-IR, and long-wave IRimages.

FIG. 7 shows an embodiment of one coded localization system 700; system700 employs single pixel detectors. In one embodiment, two imagingchannels are mounted in such a way that the boresight (or on-axis)directions are all parallel. The two pixels 720 have overlapping fieldsof view 702 that form intersecting contours 708. Contours indicateconstant detected values as a function of object position. This enablesestimation or detection of the location of an object using only twopixels. The fewer number of pixels required to perform a given taskresults in lower power and size. The location of an unknown object pointin the field of view has an uncertainty between locations ‘a’ and ‘b’ inthe intersecting contours 708. In one embodiment, a priori knowledgeexists to differentiate between locations ‘a’ and ‘b’. In anotherembodiment, the a priori information includes at least one of ahemisphere, intensity, wavelength, and a polarization complex value toprovide discrepancy between ‘a’ and ‘b’. Without a priori information athird channel may be used as described below.

In one embodiment, overlapping fields of view 754 of three imagingchannels, as shown in system 750, are mounted such that the boresight(or on-axis) directions are parallel with overlapping fields of view754. The on-axis positions of systems 0 and 2 are shown as 758. Anunknown object point is located between these three channels. In anembodiment the relative distance to the unknown point is given by threenormalized radius values, one for each imaging channel, r0, r1 and r2.If the normalized radius values are known for each channel, and theimaging geometry of all channels is also known, then the 3D location ofthe object point can be determined as ‘b’ or 756 in system 750. If onlysystems r0 and r1 were used an ambiguity would remain with point ‘a’ or752. In another embodiment the three imaging channels are localizationcoded so that the relative radius of each object point can bedetermined, independent of scene bias, and noise from the collection ofimaging channels and the corresponding 3D object location can beestimated. With localization coding the estimation precision is smallerthan the pixel size even if locations of the pixels or sensors and/oroptics in 3D are not well known, or the uncertainty in spatiallocalization is less than the area of a detector pixel. Multipleconfigurations are possible, for example single pixel sensors andmultiple measurements are formed through spatially displaced systems, orsingle pixel sensors and multiple measurements are formed throughtemporally displaced systems. Also coded localization systems where thecoding is both spatial and temporal provide benefit where the signalvaries in both time and space. For all of these configurations, a methodand apparatus are provided for relative object location estimation withprecision high compared to the number of pixels.

FIG. 8 shows an embodiment of coded localization system 800 using apriori information. In one embodiment, the object or optical data of theword “GENERIC” is known in advance as 810 (and 810 may be created byknown prior art). From this information a collection of knownmeasurement vectors 820 is formed and later used to test against theactual sampled measurement vectors 830 in order to perform at least onetask of detection, localization, ranging and multi-channelcorrespondence. If 810 is an image, then the number of samples forming810 can be much larger than in the sampled or known measurement vectors820 and 830. In one embodiment the vector 820 contains reduced datacompared to the source 810. In one embodiment the formed measurementvectors 820 and the sampled measurement vectors 830 can be normalizedand so form a vector metric for sub-pixel localization of the generallycomplicated target 810. The actual sampled measurements 830 are comparedat a vector level to the modeled vectors 820 and a vector normcalculated as a matching metric using processor 870. In one embodimentthe measurement vector is compared to a stored or estimated objectvector to detect at least one of the object, object orientation andobject location. In another embodiment the measurement vector is usedwith a mathematical model of the object to estimate object parameters.

The system of FIG. 8 applies to an “object camera” or “object sensor”.That is, in a large number of applications information is only desiredabout certain type of objects, or certain classes of objects. Someapplications may only be interested in capturing information aboutlocation and dimensions of telephone poles. Other applications may onlybe interested in the number of locations of people. All other collecteddata that doesn't represent information about either telephone poles orpeople is not really wanted and could be discarded as a pre-processingstep. In one embodiment of the coded localization system 800 (operatingas an object sensor), a model of a set of objects 810 is collected andthen reduced to the possible known measurement vectors 820. The sampledmeasurement vectors 830 are compared to the known measurement vectorsand objects in the set of application-specific objects are thendetected, and information about these objects further detected orestimated, such as their orientation, location, position, etc. Inanother embodiment human viewed images corresponding to regions aboutthe detected object can then be formed as shown in FIG. 23 from thesampled or known measurement vectors.

Even with very noisy sampled data the sampled measurement vectors canclosely match the modeled vectors and the sub-pixel location 880 of theobject 810 is detected. In an embodiment the normalized vector methodenables an invariance to object gain, as the vectors capture directionand relative magnitude. In another embodiment the normalized vectormethod enables a low sensitivity to additive noise as the process isbipolar. In another embodiment the normalized vector method enables anerror magnitude as well as direction for fast detection and acombination of detection and estimation steps. In an embodiment, amethod and apparatus are provided for context-dependent optical datareduction and storage.

FIG. 9 shows an embodiment of coded localization system 900 for motilityand motion observation and discrimination. This localization system 900is useful to build devices for localization, navigation and orientationas described in later paragraphs and figures below. In one embodiment ofsystem 900 the arrangement of arms 910, 920 and 930 may be for examplecoplanar with axes Δx and Δy and orthogonal to Δz in referencecoordinate system 980. Reference coordinate system 980 is also aninertial frame of reference for motility and mobility estimates. In anembodiment of system 900 each arm 910, 920 and 930 forms motionestimates 912 as [dx1, dy1], motion 922 as [dx2, dy2], and motion 932 as[dx3, dy3], respectively for each arm, along and orthogonal to thedirection of the arm. In another embodiment, arms 910, 920 and 930 alsoprovide details D1, D2, and D3 for each of the motion estimates 912,922, and 932, respectively. In one embodiment details D1, D2, and D3 areat least one of an estimate of signal quality and noise levels,structure information such as polarization state and wavelengthintensity variance, object specific features and energy information suchas spatial frequency modulation and intensity across field. In oneembodiment the coded nature of the coded localization system 900provides details with an improvement in at least one of SNR, spatialresolution, size, weight, and power consumption. In another embodimentthe coded nature of the coded localization system 900 provides motionestimates 912, 922, and 932 with an improvement in at least one ofprecision, motility detection, spatial resolution, and accuracy. Inanother embodiment calibration to a reference provides an improvement inabsolute orientation and localization. In an embodiment, a method andapparatus are provided for absolute orientation estimation. In anembodiment, a method and apparatus are provided for absolute 3D locationestimation. In an embodiment, an improvement is provided for orientationand 3D location estimation for positioning compared to prior artinertia-based navigation systems. In an embodiment, system 900 providesa method and apparatus for performing at least one of orientation and 3Dlocation estimation for indoor applications. In an embodiment, system900 provides a method and apparatus for performing at least one oforientation and 3D location estimation for applications with unreliableGPS coverage.

In subsystem 990 the orientation of the motion indicated by arm 922 canbe decomposed at any instant in time using angle β into components [δx2,δy2]. Also in subsystem 990 the orientation of the motion indicated byarm 932 can be decomposed at any instant in time using angle α intocomponents [δx3, δy3] of the overall reference coordinate system 980 andcombined with [dx1, dy1]. In subsystem 990 the angle θz is rotation inthe Δx Δy plane, or rotation about the Δz axis. In an embodiment ofsystem 900 the true motion and motility of each arm 910, 920 and 930 inreference coordinate system 980 are coupled based on a frame formed byarms 910, 920 and 930. In another embodiment of system 900 the motionestimates 912, 922, 932 are independent of each other and so a number ofuseful relationships and constraints between the independent motionestimates and the coupled constraints of the geometry in system 900 withrespect to reference coordinate system 980 can be established.

In one embodiment when θz=0 and dx1=0, δx2=0 and δx3=0, when dy1>δy3 andδy2=δy3 reference coordinate system 980 is pitching. In anotherembodiment when θz=0 and dy1=0, δy2=0 and δy3=0, when dx1=0 and δx2=−δx3reference coordinate system 980 is rolling. In another embodiment whenθz=0 and dy1=0, dy2=0 and dy3=0, when dx1=dx2=dx3 reference coordinatesystem 980 is yawing with proportion to dx1.

In an embodiment a non-planar surface 982 can be profiled in at leasttwo dimensions using a fixed reference coordinate system 980. In anotherembodiment of system 900 a moving surface or portion of surface 982 canbe detected and estimated based on violations of the constraintspresented by the physical arrangement of the arms 910, 920, and 930. Inanother embodiment of system 900 a known planar or non-planar surface982 can be utilized to estimate the geometry of the physical arrangementof the arms 910, 920, and 930, for example during a calibrationprocedure. As such system 900 can estimate motility and relative motion,and also detect and estimate external motion. In one embodimentmeasurements are made in at least one of spatial, temporal, spectraland/or polarization domains. In another embodiment measurements are madein at least two of spatial, temporal, spectral and/or polarizationdomains.

In an embodiment system 900 may contain electromagnetic energyconverters 997 for converting energy 995 into electrical signals whichmay be further digitized by at least one of electromagnetic energyconverters 997 and process 998 and a physical media 996 for transportingelectromagnetic energy to the electromagnetic energy converters 997.Converted data is processed by process 1998 to form a sparse set of N+1motion estimates 999 denoted d0, d1, . . . , dN. Electromagnetic energy995 may be in the form of ultraviolet to visible to long wave infraredwavelengths, acoustical wavelengths and radio wavelengths for example.In an embodiment electromagnetic energy 995 may be further modified bythe physical media 996 prior to detection for example to affect at leastone of a polarization state, a wavelength, a spatial intensity and amodulation of a spatial frequency. In another embodiment physical media996 may also impart a variation between modifications to electromagneticenergy 995 among the arms 910, 920, and 930 to affect at least one ofthe optical properties of magnification variation, field of view,optical axis skew, field intensity variation, field polarizationvariation, and field aberration content.

In an embodiment of the invention disclosed herein a method for designof coded localization systems included maximizing the information for agiven task. Imaging systems are often designed to produce visuallypleasing pictures. In cases when the systems are producing information,design methods related to information are required. A preferred methodis system design that minimizes the Cramer-Rao bound and maximizes thecorresponding Fisher information for particular aspects of the 3D sceneinformation. In one embodiment coded localization systems are designedso that the cross-information between channels sampling the same remoteobject is reduced compared to no coding. In another embodiment the sumof the output of coded imaging channels is essentially independent ofthe independent variable or variables in the domain of the codes.

Design of these coded systems can be understood through informationtheory. Designing via methods disclosed herein can maximize the possibleprecision of estimated parameters, or more generally, increase theinformation capacity of the coded system relative to the objectparameters of interest.

The variance of the best unbiased estimator θ′ of a deterministicquantity θ, based on noisy measurements, is bounded by the Cramer-Raobound. Or,

Var(θ′)≦Cramer-Rao Bound=J(θ)⁻¹=inverse of the Fisher Information matrixJ.

The Fisher Information matrix J is a fundamental matrix related to theoverall system that describes the sensitivities of the system to theexpected unknowns. Assume the measurements y are defined by a set ofunknowns. Then the elements of J are products of the partial derivativesof the measured signals with respect to the unknowns. In the additiveGaussian noise case the Fisher Information matrix can be written as

$J_{ij} = {\frac{d{\underset{\_}{y}}^{T}}{d\; \theta_{i}}R^{- 1}\frac{d\underset{\_}{y}}{d\; \theta_{j}}}$

where the vector y represents the measurements that are parameterized bythe vector θ. The matrix R is the correlation matrix of the additiveGaussian noise. If the noise is uncorrelated and identically distributedthen the correlation matrix is the identity matrix multiplied by thenoise variance. In one embodiment a diagonal correlation matrix for thefollowing results is assumed. In another embodiment variations based onnon-Gaussian and non-identical noise statistics can also be followed forsystems that are not well modeled by this type of noise. In low-noisecases many types of systems are well modeled by additive Gaussian noise.The Fisher Information matrix can also be written as

$J_{ij} = {\sum\limits_{\# \mspace{14mu} {of}\mspace{14mu} {channels}}\begin{bmatrix}{{info}\mspace{14mu} {on}\mspace{14mu} x_{o}} & \cdots & {{Xinfo}\mspace{14mu} {on}\mspace{14mu} x_{o}} \\\vdots & \ddots & \vdots \\{{Xinfo}\mspace{14mu} {on}\mspace{14mu} x_{o}} & \cdots & {{info}\mspace{14mu} {on}\mspace{14mu} x_{n}}\end{bmatrix}}$

where the sum is over the different channels. The desired information isthe parameter x_(o). The Fisher Information matrix contains entriesrelated only to the system information about x_(o), as well as what'scalled cross-information about x_(o) and the other unknowns, as well asinformation about the other unknowns. The cross information on x_(o) canbe considered a nuisance parameter as its presence negativelycontributes to estimating the value of x_(o). Or, the nuisanceparameters increase the related Cramer-Rao bound on the parameter x_(o).In one embodiment of multi-channel system design a goal is to maximizethe overall system information about a particular parameter while alsominimizing the cross information related to the same parameter. Thiswill result in a system optimized to estimate the chosen parameters fromthe crowed object information space.

In one embodiment a simplified object model such asy_(i)(x)=Gh_(i)(x−x_(o)), with gain G and position x_(o) unknown, andwith uncorrelated white noise is employed. The partial derivatives ofthis model are

${\frac{d\mspace{14mu} {yi}}{d\mspace{14mu} G} = {h_{i}\left( {x - x_{o}} \right)}},{\frac{d\mspace{14mu} {yi}}{d\mspace{14mu} x_{o}} = {{- G}\mspace{14mu} {h_{i}^{\prime}\left( {x - x_{o}} \right)}}}$

where h′(x) is the derivative of h with respect to x. The FisherInformation for one channel is then given by

$J = \begin{bmatrix}{h_{i}\left( {x - x_{o}} \right)}^{2} & {{\,^{-}G}\mspace{14mu} {h_{i}\left( {x - x_{o}} \right)}\mspace{14mu} {h_{i}^{\prime}\left( {x - x_{o}} \right)}} \\{{- G}\mspace{14mu} {h_{i}\left( {x - x_{o}} \right)}\mspace{14mu} {h_{i}^{\prime}\left( {x - x_{o}} \right)}} & {G^{2}\mspace{14mu} {h_{i}^{\prime}\left( {x - x_{o}} \right)}^{2}}\end{bmatrix}$

The upper right and lower left hand quantities are the nuisanceparameters for the quantity x_(o). With multiple imaging channels andtwo unknowns, minimizing the Cramer-Rao bound on x_(o) is equivalent todiagonalizing the Fisher Information Matrix J. This happens when

${\sum\limits_{{Channel}\mspace{14mu} \# \; i}{{h_{i}\left( {x - x_{o}} \right)}\mspace{14mu} {h_{i}^{\prime}\left( {x - x_{o}} \right)}}} = 0$

In one embodiment, with two channels and the above imaging model J isdiagonalized when h₁(x)=sin(wx), h₂(x)=cos(wx), for a spatial period w.In another embodiment the form of the individual channel codes can be asum of cosines where the frequency of each cosine is a harmonic of afundamental period. The phase of the sum of cosines for each channel isdesigned so that the Fisher Information is diagonal. In anotherembodiment the form of the individual channel codes can be a sum ofproducts of cosines where the fundamental frequency of each cosine is aharmonic of a fundamental period. The phase of the sum of products ofcosines for each channel is designed so that the Fisher Information isdiagonal.

Incoherent imaging systems are non-negative. One embodiment ofnon-negative code is: h_(i)(x)=0.5 sin(wx)+0.5, h₂(x)=0.5 cos(wx)+0.5,for a spatial period w. By phasing of multi-channel sinusoidal codesestimation performance can be equivalent to “spatially coherent”systems. In another embodiment the sum of cosine and sums of products ofcosine codes also has a bias term in order to make the codenon-negative. The non-negative requirement forces the amplitude ofindividual sinusoidal components to decrease in multi-sinusoidal codemodels. This has the effect of reducing the information capacity relatedto those particular code components.

FIG. 10 illustrates graphical data from a 3 channel coded system thatincreases the Fisher Information on object location. Graph 1010 is froma system of three unipolar or non-negative channels. Graph 1030 is froma system of three bipolar channels, or the channels from 1010 butwithout the bias term. The additive noise is considered to be the samefor both the biased and unbiased systems. The equations of the sinusoidsare given by cos(2pi x+phase), where x ranges from −0.5 to +0.5. Thephase of the three sinusoids are 30 degrees, 150 degrees, and 270degrees, which are 2pi symmetric with a 120 degree separation.

From graph 1030 the unbiased channels are bipolar (have negativevalues). While a temporally incoherent optical system cannot havenegative measurements, the 3 channel biased system of 1010 can haveidentical information about the unknown position variable x as theunbiased system. The Fisher Information related to measurements 1010 isshown in 1020, and for 1030 is shown in 1040. These Fisher Informationplots also include an unknown additive bias term. With proper systemdesign the addition of unknown bias does not negatively affect thecaptured information about the desired parameter or parameters. Thevalue of information about variable x is the same in 1020 and 1040. And,the cross information related to X is zero for both 1020 and 1040. Thenon-zero cross information curve in 1020 is related to the crossinformation between G and B.

FIG. 11 shows graphical data 1100 for a coded localization systemperforming sub-pixel information gathering, detection and estimation ofa 1D line on a set of coded pixels. The set of coded pixels, the codes,and the disparity of the codes constitute a “system” as a functionaldescription used hereinafter. The top row shows the aerial images of aline as it would appear during one period of the codes before sampling.In one embodiment the period of the codes is over one or less pixels. Inanother embodiment the period may be over many pixels. In one embodimentthe pixel codes are biased sinusoids and cosines and the measured datais the inner product of the individual codes with the aerial image data.

In one embodiment the imaging model is unknown gain, bias and verticalline location and 4 codes record each of the horizontal or verticalsub-pixel dimensions of the line. In an embodiment when the 4 codes aresinusoids phased by 90 degrees the difference between the two 180 degreeout of phase channels is a bipolar sign with no bias. The two sets ofdifferences yield two bias-free measurements that are together shiftedin phase by 90 degrees. These two measurements may represent the realand imaginary part of the Fourier Transform coefficients at one spatialfrequency of the sub-pixel object. The complex representations of thesedifferences of Fourier transform measurements are represented as thehorizontal and vertical vectors in the row of unit circles shown in FIG.11. The two vectors have been scaled so that the largest magnitude isone, and the vector with the smaller magnitude is scaled proportionatelyto one. In the case of imaging a vertical line, all the vertical vectorshave essentially zero magnitude. The horizontal vectors directlyindicate the sub-pixel location of the vertical line.

In another embodiment when the gain and bias were known a priori thenonly 2 codes phased by 180 degrees are needed for each vertical orhorizontal direction of the object. In one embodiment the a prioriknowledge is obtained from an un-coded channel. An example of such codesis given in system 1150 with graphs 1180 and 1190. Graphs 1180 and 1190describe a “system” as they show the system design aspects (the codevalues), with two different codes as the system requires, and theassociated Fisher information, that enables the results shown in 1100.Graph 1180 in FIG. 11 shows two sinusoidal codes that have 2pi symmetry,or are phased by 180 degrees. A bias has been added so that theamplitude of the codes is non-negative. The Fisher Information onrelated to these codes is shown in graph 1190. Notice that the maximumvalue of Fisher Information related to spatial position x is about ⅔that of the 3 channel coded system of plot 1020 of FIG. 10. From graph1190 the information related to spatial location is biased as a functionof sub-pixel location. The system designer can accept this bias orredesign the 2 channel code to form a compromise between bias and 2pisymmetry. In a comparison case, FIG. 25 shows two examples of twochannel codes that illustrate other choices. In one embodiment thechannel codes in FIG. 25 are polynomial codes.

The two channel codes 2500 of FIG. 25 have 2pi symmetry (or the sum ofthe codes are constant as a function of spatial position) as shown in2510. The square root of the inverse of the Fisher Information, orCramer Rao Bound (CRB), for this system is described in 2520. The CRBfor this system is also biased but has less change than does theequivalent CRB for the two channel system 1150 in FIG. 11. System 2550in FIG. 25 describes another related design that does not exactly hold2pi symmetry but does achieve an unbiased CRB, and therefore an unbiasedFisher Information as shown in 2560. The collection of graphs in 2550 isconsidered a “system” since it describes the design of the codes, theorthogonality of the code designs, and the Fisher information achieved.Each code taken alone is unimportant, rather it is theinter-relationship of the codes or the system-level accomplishment ofthe combination of codes that is important. There are numerous other twochannel codes such as those in FIG. 11 and FIG. 25 that balance the needfor orthogonality, non-negativity, 2pi symmetry and CRB/FisherInformation bias.

FIG. 12 shows a phased sinusoidal coding for a localization system 1200and responding to a sub-pixel edge. The edge is a vertically alignededge so the vertical measurement vectors have zero magnitude. Thehorizontal vector, being the Fourier Transform at one spatial frequencyof the sub-pixel imagery, shifts with the location of the edge. Thelocation of the sub-pixel edge location can be directly measured fromthe horizontal measurement vector. In one embodiment system 1200 uses 4phased channels per direction with no a priori knowledge of the edgegain and bias. In another embodiment system 1200 uses 2 phased channelsper direction with a priori knowledge of the edge gain and bias.

In another embodiment the number of actual unknowns in the image data islarger than the unknowns that are actually desired. The image gain andbias, for example, are two quantities that are often considered lessimportant in 3D imaging/ranging than the range to particular spatialfeatures. In an embodiment to further simplify and reduce system size,weight and power, hi-resolution imagery is acquired and used to modelthe sampled measurement vectors a priori. With this high resolutioninformation the gain and bias can be considered known. In anotherembodiment high-level and high resolution image processing features maybe used to guide the image processing.

In another embodiment a high resolution imaging system is equipped withcoded localization systems that provide measurements in 3D that are usedin conjunction with the high resolution images to augment disparity inmulti lens camera systems. In another embodiment the augmented disparityis used to generate coordinates for at least one of entertainment,mapping, navigation, survey and geoinformation systems.

In another embodiment an estimate of bias is formed through the sum ofthe outputs of the measurement channels. The sum of the measurementchannels can be an accurate measure of the bias (code bias plus the biasfrom the object, system, etc.) if the codes are designed with 2pisymmetry. An important characteristic of 2pi symmetry is that the sum ofthe measurement channels can be a constant for a particular object inthe domain of the code. Re-consider the 3 channel measurement system of1010 from FIG. 10. The sum of the codes or measurements 1010 is aconstant as a function of normalized location. The sum of the code thenprovides no information about location, only information about bias.

In a method for designing a 2D optical system and a system PSF, codesdesigned to have 2pi symmetry are those where the sum of the measurementchannels is independent of the particular domain variable, be itspatial, polarization, temporal, etc. Consider spatial coding. Then aset of codes has 2pi symmetry if the combination of PSF and spatialcodes, when all channels are summed, produces a sampled value that isideally independent of spatial location. A multi-aperture 2×2 systemwith 2 horizontal and 2 vertical sinusoidal codes can have 2pi symmetryif the sinusoids for each channel are out of phase by 180 degrees.Design of codes also depends on field of view arrangements of multiplechannels including overlap and adjacency, or system field of view(SFOV), as well as the task at hand (where is that, and where am I),where codes that have overlapping fields of view or specialized SFOVscan provide multi-channel signals with more information than independentand un-coded sensors when detecting objects or motion and alsodiscriminating between global and local motion.

FIG. 13 is an embodiment of a coded localization motility and motionobservation and discrimination system 1300. In one embodiment of system1300 each arm 1340, 1350 and 1360 forms motion estimates 1342, 1352, and1362 respectively, within overlap regions 1392 formed by the overlap oflarge region 1364 with smaller regions 1344 and 1354. In an embodimentthe motion estimation 1342, 1352, and 1362 also contain details D4, D5,and D6 are for example independent estimates of wavelength intensity. Inanother embodiment details D4, D5 and D6 contain polarization intensityand angle.

For one embodiment of system 1300 the true motion and motility of eacharm 1340, 1350 and 1360 in reference coordinate system 1380 are coupledbased on a frame formed by arms 1340, 1350 and 1360 overlap regions1392. In another embodiment of system 1300 the motion estimates 1342,1352, 1362 and details D4 and D5 are independent from each other. Inanother embodiment the details D6 from region 1364 have a partialdependence on regions 1344 and 1354.

In one embodiment system 1300, when rotated about the z axis, has asystem field of view (SFOV) consisting of sparse region 1393 pannedacross a region (indicated by arrow 1395) such as sky, resulting in aSFOV shaped like a band with a finite extent of elevation and 360degrees in azimuth. Features 1394 are in one embodiment at least one ofa measure of polarization strength and direction. Features 1394 are inanother embodiment a response to polarization codes from the codedlocalization systems. In one embodiment various parameters of thefeatures 1394 are reported in the details D4, D5 and D6. In anotherembodiment details D4, D5, and D6 contain at least one of parametersrelated to contrast, intensity contours, wavefront slope, and wavelengthinformation.

In another embodiment system 1300 provides details D4, D5, and D6 forregions contained within the SFOV where the details contain polarizationfor sky for at least a portion of the SFOV. The pattern of vectors inthe at least one portion of the SFOV provides geocentric azimuth,elevation and roll estimates from polarization patterns in the sky. Inan embodiment motion and motility estimates of the reference coordinatesystem 1380 such as motion 1395 across pattern 1394 allow the portionsof the SFOV to be mapped into an adaptive sky model. In an embodimentthe adaptations include influences from near-horizon effects includingat least one of pollution, dust, smoke, and thermal gradient wavefrontdistortion.

While measuring orientation of signals containing partial polarizationis common and well known, measuring signals with polarization parameterswith very low degree of partial polarization is not well known. In oneembodiment of measuring low degrees of partial polarization the unknownmedium of FIG. 4 is included in system 1300 and acts to change thedegree of polarization, often to a very low value that can be recoveredby use of the invention disclosed herein. In another embodiment theunknown medium of FIG. 4 is at least one of clouds, fog, thermalgradient wavefront distortion and pollution.

In another embodiment the collection of optics and detector of system1300 have some particular field of view (FOV), which may be a portion ofthe overall SFOV, and instantaneous field of view (iFOV) and forms Nmeasurements. The iFOV is the FOV of a particular pixel. In anotherembodiment the iFOV may be a function of the particular pixel, throughfor example, distortion of the optics. In one embodiment codedlocalization systems are disclosed where the sampled PSF of the imagingsystem is space-variant. In another embodiment the N measurements form ameasurement vector as a function of spatial position. In anotherembodiment the N measurements form a measurement vector with a magnitudeand phase as a function of spatial position. In another embodiment the Nmeasurements are configured to be unbiased. In another embodiment the Nmeasurements are designed so that the sum of the squares of the Nmeasurements is constant in the absence of noise.

In one embodiment the system 1300 produces the data matrix M in FIG. 3through the unknown medium 480 in FIG. 4. Assume that (x1,y1) is thespatial location of the data in M. Then the collection of data through apolarization dimension is M(x1,y1,i), i=1, 2, . . . N. For the momentassume that the unknown medium 480 in FIG. 4 as applied to system 1300just imparts a constant magnitude or gain and bias over spatial,spectral, temporal and polarization domains of the measured data. Inthis embodiment of data M the measured data in the polarization domaincan be described as

M (x1,y1,i)=G cos(wi+phi1)+Bias

M (x1,y1)=G cos(wi+phi1)+Bias

where M is a vector of M and G is an amplitude dependent on the signal493 a-c in FIG. 4, the unknown medium 480 attenuation and thesensitivity of the sensing system 1300. The phase phi1 is related to therelative orientation of the signal and the sensing system. The Bias isalso related to the signal, the unknown medium and the sensing system.The term wi is considered known and is related to the rotation of thecoding elements 340 to 340 n in FIG. 3 and coding elements 440 to 440 nin FIG. 4 when the coding elements form a polarization analyzer. In anembodiment the degree of partial polarization can be given by the ratioof G to the Bias. In another embodiment orientation is determined usingG and phi1. In another embodiment data M is acquired with minimized Biasand maximized G. In another embodiment data M is acquired with equalBias and G.

In real systems the sensing of M is always in the presence of noise. Theestimation of G, phi1 and Bias is then a statistical estimation problem.The best possible estimation performance can be described in terms ofFisher Information and Cramer-Rao bound for this particular embodiment.In an embodiment design of the sensing system 1300 involves optimizingthe system to maximize the Fisher Information relative to the quantitiesof interest while reducing the cross information with other quantities.This is equivalent to reducing the Cramer-Rao bound for the bestpossible unbiased estimation performance of particular parameters of thesystem. In one embodiment the parameters to be estimated include atleast one of a linear polarization angle and magnitude.

In one embodiment the Fisher information matrix is created by the innerproducts of the partial derivatives the data model M with respect to thesystem unknowns G, phi1 and Bias. Or

J(i,j)=[dM/d(theta_(i))]^(T) [dM/d(theta_(j))

If di=cos(w*i), i=1, 2, . . . N is chosen so that the sum(d)=0 then onlythe diagonal elements of the Fisher information are non-zero and

J(1,1)=N/2,J(2,2)=NĜ2/2,J(3,3)=N

In one embodiment N=3 and the different channels of the coding elements340 to 340 n in FIGS. 3 and 440 to 440 n in FIG. 4 are linear polarizersand are rotated consecutively by (180/3) degrees (and the effect on themeasurement phase is 2×180/3) degrees then w*i=0, 2*pi/3, 4*pi/3. Andsum(sin(w*i))=0. The best possible unbiased estimation variance of the 3parameters are then given by

var(G _(est))>=2 sigmâ2/N

var(phi1_(est))>=2 sigmâ2/(Ĝ2N)

var(Bias_(est))>=sigmâ2/N

where sigmâ2 is the variance of the additive white Gaussian noise. Inmany situations this noise is dominated by shot noise which isunavoidable even in “perfect” detectors. In one embodiment shot noise isminimized by adjusting exposure time to minimize Bias while maximizingG. In another embodiment shot noise is minimized by adjusting sensorsensitivity to minimize Bias while maximizing G. In another embodimentshot noise is balanced with read noise by adjusting sensor sensitivityto equalize Bias and G.

In one embodiment the design the coded localization system will choosethe channels of the polarization domain so that the cross informationbetween unknowns is zero or as low as possible while also acting toincrease the Fisher Information of the desired unknowns. In anotherembodiment the unbiased codes sum to zero. In another embodimentunbiased codes form an orthogonal set.

In practice the amplitude of the signals 493 a-c in FIG. 4 can be verysmall relative to the noise level of the sensing system 1300. High noiselevels lead to a reduction of localization precision. As an example, thesignal amplitude in the polarization domain can be less than 1% of thefull scale signal. With a ten bit detector operating close to full well,the amplitude of the detected signal can be about 0.01*1024 or about 10counts out of maximum of 1024 measured grayscale counts. With even aperfect detector the standard deviation of the shot noise would be aboutsqrt(1024)=32 counts resulting in a signal-to-noise level far less thanone. FIG. 14 is an embodiment of a coded localization system showing 8polarization steps with 2pi symmetry, or pi/4 phase steps, or pi/8 realrotation steps of a linear polarizer. In one embodiment to improve theestimate of the polarization signal amplitude and phase morepolarization steps can be used. FIG. 14 shows graph 1410 with 8polarization steps with 2pi symmetry, or pi/4 phase steps, or pi/8 realrotation steps of a linear polarizer. Graph 1420 in FIG. 14 shows theresult of this signal in shot noise.

The use of more than the minimum number of polarization phase samples isuseful for detecting when the actual measured signal does not correspondto a sinusoidal model. With proper choice of optics and detector theiFOV can be made small compared to the angular extent of the signals asseen by the sensing system. When the iFOV is smaller than the signalangular extent a larger number of samples can be used in the estimationof the unknowns thereby increasing estimation precision in unavoidablenoise.

If the system is designed to minimize the cross information spatialresolution can be traded for estimation precision. If the signals havelower spatial resolution than the sensing system this can be abeneficial trade. In one embodiment the received data can be configuredsuch as:

[{right arrow over (M)}(x1,y1; M (x2,y2); M (x3,y3); . . . ; M (xN,yN)]

where the concatenation of noisy sinusoid is seamless due to the 2pisymmetry of the designed system. In another embodiment the received datacan be configured such as:

1/N*ΣiM (xi,yi )

FIG. 14 graph 1430 illustrates an embodiment for spatial concatenationof 2pi symmetric polarization samples to increase the ability toestimate signal phase in noise. Any one imaging channel is representedby graph 1420. Graph 1430 shows the polarization samples in 4×4 spatialneighborhoods with additive shot noise. Graph 1440 is the FourierTransform of graph 1430, plotted as 20*log 10 of the Fourier Transformof the data. While graph 1430 can be considered roughly a sinusoid graph1440 clearly shows the sinusoidal nature of the signal indicated aspeaks 1445 in the spectrum. The phase of the signal at the signal peaks1445 can be estimated from the phase of the complex values of theFourier Transform.

From the above Cramer-Rao bounds, the variance of best possible unbiasedestimate of the signal phase is dependent on the amplitude of thesignal. As the signal amplitude decreases the ability to estimate thephase also decreases, or the variance of the best unbiased estimator ofsignal phase increases. In an embodiment the processing required toestimate the phase of unknown polarization signals can change as afunction of the signal amplitude or degree of polarization. In contrast,as the signal bias increases so too does the noise. In an embodiment theprocessing required to estimate the phase of unknown polarizationsignals can change as a function of at least one of the signal bias anddegree of de-polarization.

FIG. 15 is an embodiment of the invention including stacking samples andperforming Fourier decomposition. Graph 1510 shows a noise-freesinusoidal signal that has been stacked as described in FIG. 4 to form along signal. Signal 1510 has a bias of 200, degree of polarization 1%,amplitude 2 with 8 polarization angles across 40×40 spatial samples withno noise. Graph 1520 shows the Fourier transform of the signal in graph1510. Graph 1530 shows the signal in graph 1510 with shot noise. Signal1530 has a bias of 200, degree of polarization 1%, amplitude 2 with 8polarization angles across 40×40 spatial samples with shot noise. In anembodiment the Fourier transform is used to estimate the magnitude andphase of the underlying signal as shown in graph 1540 as peaks 1545. Thesinusoids are clearly above the noise in the spectral analysis.

FIG. 16 is an embodiment of the invention including stacking samples andperforming Fourier decomposition. Graph 1610 shows a sinusoidal signalthat has been stacked as described in FIG. 4 to form a long signal. Thedifference between 1610 and 1530 is that the exposure related to 1610has been reduced by approximately a factor of 10 compared to 1530. Thisreduction in exposure reduced the signal bias and shot noise but not thesignal amplitude. Signal 1610 has a bias of 20, degree of polarization1%, amplitude 2 with 8 polarization angles across 40×40 spatial sampleswith shot noise. Graph 1620 shows the Fourier transform of the signal ingraph 1610. The sinusoids are clearly above the noise in the spectralanalysis. Even with fewer samples as in graph 1630, the spectralcomponents are still clearly visible in graph 1640 as points 1645. In anembodiment the optics reduce the memory storage required compared to theFOV and localization uncertainty. In another embodiment optics anddata-dependent signal processing reduces the amount of data storedcompared to classical imaging systems.

Due to the non-uniformity of the unknown medium 480 described earlierthe SNR can vary spatially across typical images. Therefore the abilityto implement a spatially varying process will further improve results,for example by correctly selecting the number N or the fundamentalexposure time to use in given FOV or iFOV. In an embodiment where theamplitude of the signal is limited but the bias can be increasing theexposure time can be optimized to equalize gain and bias. In anembodiment arms 1340, 1350, and 1360 in FIG. 13 have different exposuretimes providing spatially varying sampling if G and Bias in the SFOV.

FIG. 17 is an embodiment of the invention showing system 1700 where theamplitude 1720 and bias 1720 are shown for increasing exposure onhorizontal axis. Exposure value 1730 can be considered an optimalexposure value since the amplitude of the signal is maximized Increasingexposure beyond 1730 only increases the shot noise in the system. Lowdynamic range signals such as these can benefit from varying theexposure as a function of detected bias and amplitude present in thesignals. FIG. 17 also shows a symbolic representation for spatiallyvarying data acquisition and also spatially varying processing. In oneembodiment the spatially varying data acquisition changes as a result ofthe estimated polarization amplitude or degree of partial polarization.In another embodiment the spatially varying processing changes as aresult of the estimated polarization amplitude or degree of partialpolarization. One goal of this spatially varying data acquisition andprocessing is to bound the uncertainty in the estimate of thepolarization phase below some level. Box 1750 shows a symbolic map ofestimated polarization intensity in spatial coordinates. There are threebroad regions of polarization amplitude H, L1 and L2. In one embodimentthe region denoted by H could be defined by the spatial regions wherethe estimated degree of polarization is >10%. Regions L1 and L2 areregions where the degree of polarization is say, 5% and 1%. In anembodiment at least one of the acquisition parameters and processingparameters within regions L1 and L2 can change in order to improve orbetter match the estimated phase uncertainty over the entire spatialextent.

Box 1760 shows another embodiment of the invention with unique spatialdata acquisition and processing regions that could be increased toimprove estimated phase accuracy. The data acquisition and processingparameters related to region H is some value, while in region L1approximately 2×2 more polarization samples are gathered and in regionL2 10×10 more polarization samples are gathered, wherein gatheredincludes at least one of collecting more data temporally and spatiallyover at least one of longer periods of time and larger spatial regionsand denser sampling periods. In one embodiment the increased number ofpolarization samples can be through larger spatial neighborhood. Inanother embodiment the increase in samples can be through at least oneof additional samples in time, through additional polarizationmeasurements and measurements in different colors and imaging channels.In another embodiment an increase in polarization samples can be througha reduced exposure and multiple exposures.

In one embodiment when the spatial extent of localized information isvery broad and has low variation, very few detector elements can beutilized to understand the information in the object space. In anotherembodiment the object space is not sparse and is highly variable and thetask is insensitive to such a signal and very few detector elements canbe utilized to understand the information in the object space.

FIG. 18 is an embodiment of a localization code that can be appliedacross the field of view of a bipolar two detector motion sensor systemsimilar to that illustrated in FIG. 5. The embodiment in FIG. 18 asapplied to the apparatus of FIG. 5 shows a system 1800 of linear codesthat vary with the particular object FOV of the system. The left channel1810 and right channel 1820 in FIG. 18 represent the response of the setof complimentary lens regions 550 as measured at the sensor 520 and 521in FIG. 5, as a function of FOV. For a finite number of lens regions 550the actual code could be a stepped structure vs. FOV and as a functionof L3 and R3 in FIG. 5. As the number of channels becomes large, or ifindividual optics for each channel have a wide FOV, the code can becomesmooth and continuous vs. FOV. For a particular moving object located atFOV_(K), the particular complimentary lens region k would then produce asignal measured be the sensor with amplitude of A(L_(K)) and A(R_(K))times some unknown gain.

In one embodiment an angle and gain normalized code represented in FIG.18 is such that the left channel 1810 is represented by x, |x|<1 and theright channel is 1−x, |x|<1. In this case the minimum FOV is representedby x=0 while the maximum FOV is represented by x=1. The measurementmodel is then L(x)=G*x, R(x)=G*(1−x) where both the gain G and angle xare unknown. The values of L(x) and R(x) are equivalent to A₁ and A₂ onFIG. 5.

In another embodiment a difference-over-sum calculation is sufficient toestimate the location of x from L(x) and R(x).[L(x)−R(x)]/(L(x)+R(x))=2x−1, which is independent of the unknown gainG. In this embodiment an estimate of object angle x is:Estimate(x)=(½)[(L(x)−R(x))/(L(x)+R(x))+1].

In another embodiment angle can be estimated as a function of time theangular velocity of an object between any adjacent positions in x can bedetermined by differentiating Estimate(x).

FIG. 19 illustrates an embodiment where at least two coded localizationsystems can be used in system 1900 to estimate object range and velocityin addition to object angle and angular velocity from object motion1905. In one embodiment system 1900 contains at least two systems whereeach system is similar to that of FIG. 5 and contains bipolar detectorpairs 1920. In another embodiment the two systems can also be combinedwithin a single optic 1910. In another embodiment the object motion 1905is at least one of constant and nearly zero and system 1900 undergoes atleast one of translation and rotation to scan the object field ofinterest.

By estimating the angle to an unknown object from two differentpositions using processor 1970, range estimates can be formedindependent of the gain of the system, intensity of the object, etc. Anobject with some motion vector 1905 traversing two complimentary lensregions A(Li) A(Ri) and A(Lk) A(Rk) produce two temporal output signals1915 i and 1915 k respectively. In an embodiment the output signals 1915i and 1915 k are amplitude coded and temporally coded. The amplitudecode is due to the code designed into the complimentary lens regions.The temporal code is due to the general optical characteristics and thedirection and speed of motion of the object. The two codes are providingcomplimentary information to processor system 1970 and so in oneembodiment can be used to further increase the precision of the rangeand velocity estimates if the baseline separation of the two systems, B,is known. In another embodiment the complimentary information can beused to directly estimate the baseline B as well as the range andvelocity of the unknown objects. In another embodiment complimentarymeasurement vectors are used to estimate change in at least one oforientation and 3D location. There is only one baseline B which willyield the corresponding amplitude and temporal codes for the unknowndistant object velocity vector as measured by the two detector outputsyielding a discriminating signal for multiple inputs. In an embodimentsystem 1900 is able to count discrete objects in motion 1905 within thefield of view of system 1900.

When using multiple systems with single, dual, or multiple pixels andoverlapping fields of view, multiple functionalities are enabledincluding determination of orientation or motility and translation ofthe system and also detection and estimation of objects and objectmotion within the system field of view. FIG. 20 shows an embodiment of acoded localization system 2000 for motility and motion observation anddiscrimination. In one embodiment of system 2000 the arrangement of arms2010, 2020 and 2030 may be coplanar with axes Δx and Δy and orthogonalto Δz in reference coordinate system 2080. In another embodiment eacharm 2010, 2020 and 2030 forms motion estimates 2012 2022, and 2032respectively, in directions orthogonal to Δz, withinpartially-overlapping regions 2014, 2024, and 2034. The overlap regionis shown as hatched region 2092 in subsystem 2090. In another embodimentthe motion estimation 2012, 2022, and 2032 also contain details D1, D2,and D3 are at least one of independent estimates of signal confidence,noise level, signal level, wavelength, wavefront phase, wavefrontamplitude, and polarization direction and polarization magnitude of thepartially-overlapping regions 2014, 2024, and 2034 and overlap region2092. In another embodiment details D1, D2, and D3 are also for exampleobject specific information including at least one of object edges,object corners, object structures, object texture and features ofinterest for an object. In another embodiment details D1, D2, and D3contain statistics about object specific information.

In another embodiment of system 2000 the true motion and motility ofeach arm 2010, 2020 and 2030 in reference coordinate system 2080 arecoupled based on a frame formed by arms 2010, 2020 and 2030 and overlapregion 2092. In another embodiment the motion estimates 2012, 2022, 2032are independent of each other and so a number of useful relationshipsand constraints exist between the independent motion estimates and thecoupled constraints of the geometry in system 2000 with respect toreference coordinate system 2080 and an overlap region 2092. In anembodiment when θz=0 and dx1=0, δx2=0 and δx3=0, when dy1=k(δy3) andδy2=δy3 the magnification of the system producing motion estimates 2012is k times that of 2022 and 2032.

In an embodiment system 2000 may contain electromagnetic energyconverters 2097 for converting energy 2095 into intensity and a physicalmedia 2096 for transporting electromagnetic energy to theelectromagnetic energy converters 2097. Converted data is processed byprocess 2098 to form a set of N+1 motion estimates 2099 denoted d0, d1,. . . , dN. Electromagnetic energy 2095 may be in the form ofultraviolet to visible to long wave infrared wavelengths, acousticalwavelengths and radio wavelengths for example. In another embodimentelectromagnetic energy 2095 may be further modified by the physicalmedia 2096 prior to detection for example to affect at least one of apolarization state, a wavelength, a spatial intensity and a modulationof a spatial frequency within the overlap region 2092. In anotherembodiment physical media 2096 may also impart a variation betweenmodifications to electromagnetic energy 2095 among the arms 2010, 2020,and 2030 to affect at least one of the optical properties ofmagnification variation, field of view, optical axis skew, fieldintensity variation, field polarization variation, and field aberrationcontent within the overlap region 2092.

FIG. 21 illustrates a coded localization system 2100 for motility andmotion observation and discrimination. In one embodiment of system 2100the arrangement of arms 2110, 2120 and 2130 may be coplanar with axes Δxand Δy and orthogonal to Δz in reference coordinate system 2180, whilearms 2140 and 2150 may be coaxial with Δz. In one embodiment each arm2110, 2120, 2130, forms motion estimates 2112, 2122, and 2132,respectively, in the directions orthogonal to Δz, within non-overlappingregions 2114, 2124, and 2134 that intersect with surface 2182. Inanother embodiment each arm 2140 and 2150 forms motion estimates 2142and 2152, respectively, in the directions coaxial to Δz, withinnon-overlapping regions 2144 and 2154 that do not intersect with surface2182. In another embodiment the motion estimation 2112, 2122, 2132, 2142and 2152 also contain details D1, D2, D3, D4 and D5. In one embodimentdetails D1, D2, and D3 are for example independent estimates of thecontrast and structure in surface 2182 and details D4 and D5 are atleast one of estimates of the polarization, intensity, contrast andstructure for regions 2144 and 2154 containing at least one of scatteredsunlight, starlight, and moonlight. In an embodiment multiple systemsare used for at least one of orientation and 3D localization. In anotherembodiment multiple systems are used to estimate change in at least oneof orientation and 3D location. In another embodiment the estimated ordetected change in orientation or location is finer than the pixelspacing.

In an embodiment of the invention described by system 2100 the truemotion and motility of each arm 2110, 2120 2130, 2140 and 2150 inreference coordinate system 2180 are coupled based on a frame formed byarms 2110, 2120, 2130, 2140 and 2150. In an embodiment the motionestimates 2112, 2122, 2132 based on surface 2182 are independent of eachother and are also independent of motion estimates 2142 and 2152. In anembodiment using a high number of independent motion estimates that arephysically dependent on a common reference coordinate system 2180 therobustness of the overall motility and motion estimate is improved.

In another embodiment system 2100 further benefits from a wide SystemField of View (SFOV). The SFOV is benefited by the motility of thereference coordinate system 2180 and in an embodiment contains thesparse sampling regions 2114, 2124 and 2134 on surface 2182 and also thesparse sampling regions 2144 and 2154. The motility of referencecoordinate system 2180 is exhibited by rotating system 2100 in θz, orrotation about the Δz axis, causing the sparse sampling to estimatemotility with [dx1, dy1], [dx2, dy2], and [dx3, dy3] and also estimateapparent far field motion as [dx5, dy5] and [dx4, dy4]. In an embodimentthe far field motion estimates and the motility estimates mustcorrespond to the rigid body dynamic constraints of system 2100. Motionestimates that are produced with a high degree of confidence asexhibited by high valued details D4 or D5, that do not match motilityparameters, indicate that there was an independent motion estimate inthe far field independent of the motility and therefore detection ofexternally moving assets is enabled for a moving and motile platform asembodied by system 2100. In another embodiment the SFOV for rotating inΔz includes the annulus formed by regions 2114, 2124 and 2134 swept in acircular arc upon surface 2182 and the horizontal horizon bands formedby sweeping regions 2144 and 2154 in a circular arc across a surfaceformed by the horizon and at least one of skylight, moonlight, andtwilight. In another embodiment the coded localization system 2100measures relative locations of other objects. In another embodiment thecoded localization system 2100 measures absolute locations of otherobjects. In another embodiment the coded localization system 2100measures at least one of a relative orientation and a relative locationof itself. In another embodiment the coded localization system 2100measures at least one of an absolute orientation and an absolutelocation of itself.

FIG. 22 is an embodiment of the invention with sampling diversity forcoded localization systems. In many coded localization systems includingmotion and motility discrimination systems that use detectors operatingwith rolling shutter exposures, motion that is fast relative to row (orcolumn) exposure times can skew the image captured and samplingdiversity is employed. In another embodiment of the disclosed inventionrolling shutter correction is provided for coded localization systemsthrough sampling diversity as illustrated in FIG. 22. For objects withhorizontal motion relative to sensor sampling when sensor rows aresampled in temporal sequence or rolling shutter sampling as in 2220 and2240 the resulting images are sheared relative to a global sampling ofthe object at a single time instant. In one embodiment the shear isknown and signal processing can be used to recover the proper imagewithout the shear. In another embodiment an object is moving across therows and towards a row sampling/reading, as in 2260, and the image ofthe object is vertically compressed. In another embodiment the objectmotion is with the direction of row sampling/reading so that the imageof the object is expanded as in 2280.

Systems 2220 and 2240 show a shearing of the formed image, while 2260and 2280 show a motion and direction dependent magnification of theobject. 2260 experiences a loss of information of the object as themagnification is reduced, while 2280 shows an increase of information asthe magnification of the object has increased. Objects travellingtowards or away from sensors with rolling shutter sampling will alsoexperience a motion dependent magnification as sampling of the objectover time will result in a time-dependent magnification due to changingobject distance.

In an embodiment coded localization sampling diversity systems 2200consist of multiple rolling shutter sampling sensors in diversegeometries. System 2210 samples rows both vertically and horizontally.In an embodiment, system 2210 may be achieved by using sensors orientedand changing the readout direction of the rows and columns. In anotherembodiment system 2210 may be achieved by using mirrors and opticalelements to reverse the effective readout directions relative to theimage of the object presented to the sensor. In another embodimentsystem 2210 may be achieved by rotating sensors and changing sensorreadout timing. System 2212 is another embodiment where the readoutdirections are not orthogonal to each other. System 2214 is anotherembodiment of coded localization sampling diversity where the sensorreadout directions are opposite each other in the vertical direction.System 2215 is an embodiment where the sensor readout directions areopposite each other in the horizontal direction and reading toward eachother. System 2216 is an embodiment where the sensor readout directionsare opposite each other in the horizontal direction and reading awayfrom each other. In another embodiment of coded localization samplingdiversity, systems 2214, 2215 and 2216 can be enabled by multiplesensors or with a single sensor that changes the direction of samplingand reading after each frame. In an embodiment, system 2218 consists ofarrays and groups of sensor readout directions are varied across a widevariety of sensors. In embodiment timing skew between sensors in systems2200 produces more diversity in row and pixel sampling.

FIG. 23 illustrates an embodiment of coded localization systems formingimages using overlapping FOV 2×2 coded localization systems as depictedin FIG. 6. Codes enhance the localization and can also enable imageformation in shorter systems. In one embodiment the locations oflocalization codes can vary from to near the lens to the sensor plane.In another embodiment the codes can include both amplitude and phase. Inanother embodiment the codes can also be composed of random or pseudorandom functions. In one embodiment multiple measurements of a remoteobject are made simultaneously in at least one domain of temporal,spectral, spatial and polarization. In another embodiment multiplemeasurements of a remote object are made simultaneously in at least twodomains of temporal, spectral, spatial and polarization. In anotherembodiment measurements are made in the visible wavelengths. In anotherembodiment measurements are made in the IR wavelengths.

A particularly useful code for a 2×2 implementation is a quadrature codecomposed of biased sinusoids. This type of code can maximize the FisherInformation of a general scene.

In one embodiment quadrature mask or coding functions Ci(x,y) placed ator near the image plane and illustrated as 2304 and 2306 are given by:

C1(x,y)=½ cos(wx+phi)+½

C2(x,y)=½ sin(wx+phi)+½

C3(x,y)=½ cos(wy+phi)+½

C4(x,y)=½ sin(wy+phi)+½

where the variable w is chosen to be equal to 2*pi/dx with dx being theside length of a pixel. In this manner the four codes are a singleperiod of a biased sinusoid with 4 varying phases. The absolute phase ofthe codes phi needs to be known for each code. The sets of sinusoidsabove in the x and y direction do not have 2pi symmetry but can still beuseful for reduced height imaging and detection. One embodiment of acode that has 2 values per pixel is given by:

${C\; 1} = {{\begin{bmatrix}0.7931 & 0.2049 \\0.7931 & 0.2049\end{bmatrix}\mspace{14mu} C\; 2} = \begin{bmatrix}0.3772 & 0.6208 \\0.3772 & 0.6208\end{bmatrix}}$

The other two codes are given by C3=C1′ and C4=C2′, where ( )′ denotesmatrix transpose. The pattern on the sensor is symbolically shown asarrangement 2308 in FIG. 23. In practice, the codes are arranged overall or nearly all pixels assigned to a particular optical channel. Theset of codes can be written as a basis set representing the system or:

${Cx} = \begin{bmatrix}0.7931 & 0.3772 & 0.7931 & 0.3772 \\0.7931 & 0.3772 & 0.2049 & 0.6208 \\0.2049 & 0.6208 & 0.7931 & 0.3772 \\0.2049 & 0.6208 & 0.2049 & 0.6208\end{bmatrix}$

The set of sampled measurements M_(s), related to all four channels withthe corresponding object points, can then be written as M_(s)=Cx[i_(1,2)i_(2,1) i_(1,2) i_(2,2)]

In another embodiment the information desired are the original sub-pixelimage values, or the data Is, for all values of the imaged scene. Noticethat I_(s) has twice the resolution of the actual physical pixels. Or,the desired data is sub-pixel imagery relative to the sampled 2×2multi-aperture imagery. In an embodiment a full resolution image isproduced based on the coded multi-aperture image samples through alinear operation of the sampled data M_(s). or I_(s) _(—)estimate=pinv(C_(x)′)M_(s), where pinv( ) is the pseudo inverse ofC_(x).

In an embodiment of the choice of the codes that make up Cx the inversecan be well conditioned. In another embodiment a quadrature codecontaining biased sinusoids is an example of a well-conditioned Cx. Ifthe mean values of the uncoded pixel samples are available, eitherthrough a priori information, through a parallel measurement, or throughan estimate based on the coded data, this mean value can be subtractedfrom the measured data M_(s). This is equivalent to using a code thathas no bias. The columns of Cx then become orthogonal and its inverse isideally conditioned giving the best performance in the lowest lightsituations. In an embodiment the codes for reduced height systems have2pi symmetry. In an embodiment an estimate of bias can be formed by thesum of the measurements M.

In another embodiment the codes in arrangement 2308 are wavelengthspecific. Arrangement 2308 can select wavelengths in one embodiment, andbe sensitive to a selection of wavelengths in another embodiment. In anembodiment the smaller images 2320 are at least one of a selection ofvisible wavelengths, near-IR, mid-wave IR, and long-wave IR wavelengths.In an embodiment the smaller images 2320 are at least two of a selectionof visible wavelengths, near-IR, mid-wave IR, and long-wave IRwavelengths.

FIG. 23 also illustrates a simulated example of using the above methodand apparatus to achieve ½ the size and weight and effectively constantimage quality. An original resolution, or single aperture, image 2310 issimulated to be imaged through a biased quadrature 2×2 codedlocalization system with sampled data 2320. Sampled data 2320 representfour channels 2321, 2322, 2323 and 2324 imaged through each of the codesC1, C2, C3 and C4 respectively. The same codes are used on each pixel ofthe measurements in a particular channel. The measurements 2321, 2322,2323 and 2324 are then all different in a unique manner determined bythe codes. A single channel image 2330 shows more spatial detail. In anembodiment, combining the 2×2 measured data using the pseudo-inverse,the reconstructed image 2340 is produced from I_(s) _(—) estimate. Thisreconstructed image has essentially the same resolution as the originalsingle aperture image 2310 but with 2×2 coded imaging channels that areabout ½ the size and weight of the single aperture system. In anembodiment, the complimentary channels are chosen in order to reduce thelength of the overall optical system. In another embodiment, codedlocalization systems provide complimentary measurements which arecombined in digital processing to form a final image with resolutioncorresponding to the total number of measurements.

FIG. 24 is an embodiment of a design method for generalized codedlocalization systems. Information theory can ideally separate systemsfrom algorithms and produce optimized designs in terms of quantifiabletasks for specific systems. A flowchart of such a design method isdescribed in FIG. 24. One particular advantage of this type of method isthat truly global optimization can be performed over a system spaceoften too complicated for even an expert to comprehend.

In one embodiment the method starts with a basic understanding of systemtradeoffs at a fundamental information level that is relevant toreducing size, weight and power while maximizing system performance ofthe particular system. Information theory teaches that ideal estimatoraccuracy (or standard deviation) is directly correlated withsignal-to-noise level. Increasing the SNR (though more signal, lessnoise, etc.) directly increases the ideal estimator accuracy anddecreases the ideal estimator standard deviation. Increasing estimatoraccuracy is the same as increase the information capacity of the system.Pixels and SNR can be traded, in an information sense, where a generalsystem with N pixels and an SNR of S can have as much information abouta distant object as a general system with N/2 pixels and 2*S SNR. In oneembodiment the number of apertures is related to the SNR as well as thesystem size and weight.

In one embodiment step 2 involves producing a software model of thegeneral parameters of the system, the general unknowns of the system(such as tolerances, temperature, etc.) and the parameters and unknownsof the object scene (such as ideal target representation, target“clutter”, etc.). The parameters are considered deterministic andcontrol the particular state of the system. The unknowns are eitherquantities to be estimated (like the number and location of specifictargets) or quantities that can, in general, affect the estimation oftargets. In one embodiment the system model produces a noise-free systemestimate of measured parameters given the set of parameters andunknowns.

In step 3 one embodiment is for a particular set of parameters, andinformation theory such as Fisher Information and the Cramer-Rao Bound,predict the performance of the ideal unbiased estimator of systemunknowns in the presence of the other unknowns. The system parametersare directly related to the actual information capacity of the systemrelative to the task-specific targets. A general goal is to find theideal parameter set that maximizes the information through comparisonand adjustment as in step 4.

In one embodiment at least one goal that this information-based designmethod can offer is the ability to tailor the particular system to theparticular modeled object scene. For example, it is well known that whenthe scene has numerous unknowns (or sometimes called clutter) theseunknowns can negatively influence the estimation of the particularunknowns of interest. To minimize this effect, the information-basedapproach designs the system such that the system is purposely in theintersection of the null space of the clutter and the signal space ofthe particular desired unknowns. A simple example of this is two closelyspaced targets. The presence of the second target can negativelyinfluence the estimation of the first target, especially if they differwidely in radiated or reflected energy. In one embodiment the optimizedinformation-based approach will attempt to orthogonalize the system,relative to the available system degrees of freedom, such that thecross-coupling of the two targets in minimized.

Steps 5-8 in FIG. 24 expand the solution by adding more parameters. Byinitially starting with a small parameter set and system complexitymodel in steps 1-4, along with an understanding of the basic principlesof information theory of the particular system, the global optimum canbe approached and followed as more and more complexity in the systemmodel is added in steps 5-8.

When the design is suitably mature the designer can insert particularalgorithms into the system in step 9. Statistical simulation of thealgorithms and system can be compared to the performance expected fromthe information-based design efforts. In practice it may often bedifficult to reach the information-based performance with an actualsystem with real algorithms, but a comparison of the statisticalperformance and the information-based performance can be used as a guideto judge the relative level of system performance compared to the ideal.

FIG. 25 is an embodiment of an optimized two pixel system using themethod of FIG. 24. Graph 2510 with 2520 is an example of the startingpoint of one example of an information-based design approach tolocalization coding systems. In graph 2510 right channel 2512 and leftchannel 2514 will be optimized for angle detection. For this example thenoise is additive uncorrelated Gaussian noise with variance equal to 1.The assumed system parameters model two single pixel channels withparallel optical axes and known baselines and fields of view. Theoptical system is designed such that the intensity output by the singlepixel channel is defined by a polynomial, where the initial polynomialhas two coefficients (0^(th) and 1^(th) order) and produces a linearresponse vs. field angle. The initial channel responses are mirrorimages of each other as seen by graph 2510.

The unknown values are assumed to be the system gain G and the targetangle. The Cramer-Rao Bound for the starting system as a function oftarget angle in shown in graph 2520. The response 2522 is biased orvaries as a function of field angle. The collection of graphs 2500 and2550 are each considered “systems” in that they embody the overalldesign features including system gain, bias, and spatial variation ofthe polynomial codes that cooperate to form the desired estimatorvariance by coding the target information in a cooperative fashion, as acohesive system.

After optimization, with the CRB on target angle as a metric, the systemof 2550 with graph 2530 results. In graph 2530 right channel 2532 andleft channel 2534 are optimized for angle detection. The angularresponses for each channel are no longer linear resulting in a lowerCRB. The value for the CRB (normalized by the unknown gain G) is about0.6 (line 2562 in graph 2560) and is also uniform across field angle andso unbiased. This value is higher than that expected from informationtheory principles of two channel coherent systems. Two channel coherentsystems should have a CRB in standard deviation that is ½ that of asingle channel system. Or the global optimum for the CRB of the optimalsolution should be 0.5*G. At this point the designer can acceptnon-optimum performance and continue adding system complexity, such aschanging the type of parameters of the measurement from a polynomialform to a sinusoidal, or reconfigure the system parameters so that theestimated CRB can reach the ideal 2 channel performance level.

While FIG. 25 was initially selected to be implemented as a twoaperture, two channel system, where each channel had a single pixel, itcould also be considered as a specialized single channel system. In anembodiment a single aperture system where adjacent pixels have theintensity vs. field angle response of FIG. 25 would be a space-variantsystem. Space variant systems are in general extremely difficult todesign due to the lack of tools. But, by considering the system as aseries of separate channels, where each channel influences only a singlepixel, a relatively simple method of designing a space-variant systemcan be made through the disclosed information-based design methods andresulting apparatus. In one apparatus that embodies the inventionspecialized space-variant systems could be constructed of at least oneof 3D index, absorption, refraction, and diffraction such that theangular responses of FIG. 25 are realized over adjacent pixels. Aparticular benefit of this system is that the space-variant system couldenable a simple mounting directly onto a sensor such that fabricationand assembly is simplified.

Localization codes can be fairly complicated depending on the particularsystem goals. In many systems the task for finding matching objectpoints from different imaging channels, or the correspondence mappingproblem, can be difficult. The codes can be tailored to both improve theability to quickly and reliably find image correspondence as well asgive high resolution estimation.

FIG. 26 is an embodiment of the invention and describes the constructionof a code that enables both narrow FOV and wide FOV localization. Thebase code 2610 is the simple linear code of graph 2510 in FIG. 25. Thissame construction is valid for any type of code. In constructions 2600,line 2610 shows a wide FOV linear code across the normalized field angle(horizontal axis) in terms of intensity (the vertical axis). A similarversion of the code but in a modulo format forms the narrow FOV code inline 2620 in FIG. 26. The narrow FOV code is helpful to aid indiscrimination or corresponding image points within small neighborhoodsbetween imaging channels. The narrow field of view code could be over 1pixel, over 4×4 pixels, 10×10 pixels, etc. The wide FOV code 2620 isused along with the narrow FOV code 2610 in order to remove ambiguityover a wide FOV and in order to improve multi-resolution processing.

In one embodiment graph 2630 describes the combination of a narrow andwide FOV codes through a multiplicative operation. In another embodimentgraph 2640 describes the narrow and wide FOV code combined throughaddition and scaling. Scaling is used to ensure that the intensityvalues are within the range of 0 to 1.

FIG. 27 is one embodiment of a 4 channel coded system that increases theFisher Information on object location. Graph 2710 is a system of fourcoded channels formed from the four non-negative measurement channels ofgraph 2730. This is similar to the system in FIG. 10 but with 3 channelsinstead of 4.

From graph 2730 the channels are non-negative due to the addition ofbias and are phased by 90 degrees to achieve 2pi symmetry. Thiscombination of signals retains the signal subspace and nuisance subspaceorthogonality as shown in graph 2740 compared to the ideal unbiasedsystem of 2720. The 4 coded channels have the same Fisher Information orCRB on object location.

Localization codes can also be used to estimate edge angle and positionin a single pixel of the 3D angular estimation system. The code for eachimaging channel can be a function of the object edge position andorientation (or angle), not just area of the pixel covered. If themeasured pixel differences can be large compared to the noise, thenpractical single pixel multi-channel edge estimation can be achieved.

FIG. 28 is an embodiment of the invention that describes edge directionestimation. Due to the orientation and localization code on eachchannel's pixels, the measured output of a pixel can be a function ofboth the edge angle and area. In system 2800 the upper row of pixels2810 is in a square alignment and the lower row of pixels 2820 is in anon-square alignment. In one embodiment the measured pixel output is afunction of the area of the pixel covered by the edge and the angle ofthe edge to the pixel. Or, pixel output can be described by some P(area,theta) for each imaging channel. This quantity can be estimated ormeasured from each channel. In another embodiment the three estimates ofpixel covered and edge angle to the pixel can be used to solve directlyfor an estimate of edge angle and position. Or, with the three pixelvalues and knowledge of the functions P(area, theta) for each imagingchannel, estimate the orientation of the edge. In another embodimentthis information can also be used to increase the estimate precisionusing image regions.

In one embodiment the implementation of localization codes can bethrough classical optics design or phase/amplitude masks. Localizationcodes can be implemented through specialized “relative illumination”profiles. These can be enabled through optics design and/orphase/amplitude masks between the optics and sensor. As the sensor pixelresponse is always a function of chief ray angle, specialized opticaldesigns that control the angle of light into a pixel can code receivedintensity vs. input angle and pixel angular response.

In another embodiment localization codes can be enabled through opticsdesign. Sinusoidal intensity falloff with field angle can be designedvia the projection of the aperture with field, the angle of the focusedspot to the image plane with field angle and the distance to the objectand image planes with field angle. Angular intensity falloff can also bedesigned as a function of the chief ray angle (CRA) and sensor mismatchacross the field. This type of design can achieve a wide range ofcosinusoidal relative illumination profiles. Design of these profilesalong with the signal processing and system effects can able acompromise between ease of implementation, SNR and estimation accuracy.

In another embodiment localization codes can be enabled throughphase/amplitude masks placed between the optics and the sensor. Maskspositioned between optics and sensor can have wide flexibility.Non-classical responses are possible with simple optical systems. Maskspositioned on or near the sensor pixels can be designed so thatsub-pixel location (for points) and sub-pixel angle and location (foredges) can be measured in the presence of noise. Lithographic masks thatare jointly optimized with the optics and sensor CRA can offer thelargest flexibility and ultra-precision lithographic fabricationprecision and low cost in volume quantities.

FIG. 29 is an embodiment of the invention for optimizing the designspace for weight and number of pixels. This type of design is particularrelevant to localization systems where SNR can be traded for the numberof pixels, which is size, weight, power and cost. Curve 2910 describesthe standard deviation of a 4 channel coded design with 2pi symmetry asa function of measured SNR and normalized angular FOV. An angular FOV of1.0 corresponds to the instantaneous FOV of a single pixel. When thestandard deviation of angular estimates is less than 0.5 then sub-pixelangular precision is possible. In one embodiment a system based on thesedesigns uses anamorphic optics and a 1D linear sensor and 4 codedchannels form a system with 2pi symmetry, like the system of 2730 fromFIG. 27. Consider the point on curve 2910 that intersects and SNR of 7.This results in an ideal standard deviation of angular resolution of0.03 pixels.

The measured SNR is a function of the illumination and range of theobject or target, the area of the collection optics, the non-idealeffects of the optics and the sensitivity of the sensor.

In a classical system the pixel size is often designed to approximatelymatch the resolution of the optics, given by (lambda/D). If the systemparameters are sufficient to yield an SNR of approximately 7 then thepixel size (or number of pixels in the same FOV, or opticalmagnification) can be adjusted by a factor of 1/0.03 or 33×. In onedimension 33 times less pixels could be required. In a 2D system withsimilar relevant parameters the number of pixels could be reduced by33*33 or approximately 1000×.

For systems with sufficient SNR the pixel size and number of pixels canbe held constant but he size of the optics reduced thereby lowering theSNR to the level required for sub-pixel localization precision.

FIG. 30 is an embodiment of a design method and trade space 3000. Thereare a large number of subspaces or degrees of freedom 3010 that can beoptimized for a given task. In an embodiment subspaces or degrees offreedom S1, S2, to Sn are for example at least two of wavelength, time,spatial extent, field of view, and exposure and sample count. The signalcharacteristics that dictate the desired tradeoff include at least oneof estimated or known amplitude and bias. The steps in determining thetradeoffs in tradespace 3010 are also shown in 3000 as a loop. Step 1 isto determine the fixed and variable degrees of freedom. In dynamicsituations in one embodiment where the sensor or field is moving timebetween samples may be fixed and at least one of exposure and spatialextent is a variable parameter. Given the set of parameters andparameter values the information captured by the sensing system isevaluated in step 2. In one embodiment processing may include the stepsof averaging and Fourier transform processing. The output of theprocessing is evaluated in step 3 for signal phase, amplitude, bias,noise and nuisance parameters. Given the state of the parameter set theresults are reported by the system in step 4 and the results are alsoused to evaluate the current parameter states. Variable parameters areadjusted as necessary. In one embodiment the spatial extent of theprocessing is adjusted to reduce the effect of nuisance parameters. Inanother embodiment the nuisance parameters consist of at least one ofclouds, trees, foliage, pollution and dust. In another embodiment theexposure is adjusted to achieve a nearly equal bias and amplitudeestimate.

FIG. 31 is an embodiment of the invention disclosing a measurementapplication. In sequence 3100 a coded localization system that canperform at least motility and motion discrimination is at initialposition 3110 and through a 3D translation and re-orientation of pitch,yaw, and roll sequence arrives at position 3115. In this embodiment thedesired output of the system during the motion is a projection of fieldsof view 3120 on the x-y plane 3130 for a downward looking samplingsystem. Graph 3150 shows the projection results of these fields of viewwith rectification. In an embodiment the motility and motiondiscrimination system estimates at least one of a pitch, yaw, roll,x-translation, y-translation, and z-translation of itself whileundergoing general motion. In an embodiment the motility and motiondiscrimination system estimates at least one of a pitch, yaw, roll,x-translation, y-translation, and z-translation while undergoing generalmotion attached to another object. In an embodiment the motility andmotion discrimination system estimates at least one of a pitch, yaw,roll, x-translation, y-translation, and z-translation of an externalobject.

FIG. 32 is an embodiment of the invention disclosing a measurementapplication. In one embodiment coded localization system 3212 is amotility and motion discrimination system with partially overlappingfields of view 3214 and is translated 3210 in view of object 3218 andsurface 3215. In another embodiment localization system 3222 is amotility and motion discrimination system with non-overlapping fields ofview 3224 and is translated 3220 in view of object 3228 and surface3225. In another embodiment the non-overlapping fields are tangent toeach other. In another embodiment the translation is at least one of aforward and reverse direction across an object of interest. In anembodiment, localization systems 3212 and 3222 move with a velocity andsense an object that is fixed by comparing the change in relative motionin the object field. In FIG. 32 in graph 3250 if the surface were knownto be of constant distance from the sensors 3212 and 3222 then it can beconcluded that the two fields of view are moving at different velocitiesto form gap 3252 in the position measurements over time. In anotherembodiment the paired fields of view 3214 and 3224 are known to berigidly related to each other and a difference in relative position 3252is a at least one of the detection of, and a measure of relative surfaceheight and a measure of absolute surface height of objects 3218 and 3228compared to surface 3215. In another embodiment object 3218 is profiledin at least on measure of height, width, depth, pitch, yaw, and roll. Ingraph 3260 the difference of the Left and Right fields of view 3214 areshown and indicate relative range change to the object 3218. It is shownthat during forward motion the right sensor detects object 3218 first intime. While the slope of the right sensor rises the slope of the leftsensor remains constant for a brief period of time. The sequence isreversed for the reverse motion as shown in the second time half ofgraph 3260.

FIG. 33 is an embodiment of the invention disclosing a measurementapplication. Operator 3320 uses instrument 3325 equipped with codedlocalization systems. The coded localization systems form at least oneof a reference field of view measurement 3328 and an object measurementfield of view 3327. In one embodiment reference field of viewmeasurement 3328 includes the magnitude 3582 and angle 3584 of thepartial polarization of the sky. In an embodiment field of view 3327captures 3D photo point information 3355. The codes operate aspreviously disclosed to enable sub-pixel measurements of objectlocations 3355 in the image 3350. In another embodiment the codesoperate to form a reference image. In an embodiment, a method andapparatus are provided for performing long range photo pointmeasurements, with high precision measured at the image plane comparedto detector pixel spacing.

In a similar measurement activity coded systems form at least one of areference field of view measurement 3368 and an object measurement fieldof view 3360. In one embodiment the coded systems localize themeasurement system using field of view 3368. In another embodiment thecoded systems localize the object of interest 3362. In anotherembodiment the information on 3362 location is used to localize anotherobject reference point 3370 by transferring the coordinates throughcoordinate transform 3365. In an embodiment, a method and apparatus areprovided to transfer at least one of location and orientation referenceinformation from a known point to a new point.

FIG. 34 is an embodiment of the invention disclosing measuring andcontrol using coded localization systems. In system 3400 codedlocalization systems form at least one of a reference field of viewmeasurement 3410 and a surface measurement field of view 3415 and 3420.Field of view 3410 provides at least one of relative and absoluteorientation and location information. Fields of view 3415 and 3420 maycooperate in one embodiment to provide preview and postview measurementsof a grading operation on surface 3430. Orientation and localizationinformation from FOV 3410 and surface information from FOV 3415 can beused in a feedback system to control or assist in control of the vehiclethe systems are mounted on. System 3450 is also an embodiment of avehicle mounted measurement system. In system 3450 coded localizationsystems form at least one of a reference field of view measurement 3460and surface measurement fields of view 3465 for measuring surface 3480.Systems 3460 and 3465 in another embodiment provide localizationinformation in the event of GPS interruption or degradation. In anotherembodiment system 3450 provides a coded localization system 3470 thatdetects and localizes at least one of a road sign, telephone pole,manhole, and overpass. In an embodiment, a method and apparatus areprovided to for data collection and data reduction for highwayinformation. In an embodiment, a method and apparatus are provided tofor orientation and 3D location estimation for machine control.

FIG. 35 is an embodiment of the invention disclosing measuring andrecording using coded localization systems. Scenario 3500 depicts asurvey of the interior and exterior of a structure. Dotted line 3502depicts the split between interior and exterior tasks, while path 3504is an entry into the structure and path 3506 is a traversal to a secondlocation within the structure. In one embodiment operator 3510 employscoded localization system 3512 while entering doorway 3514. The codedlocalization system measures the translation and orientation of theoperator relative to the surrounding environment providing a relativemotion and orientation vector for mapping the operator movements. Inanother embodiment the coded localization system measures at least oneof the dimensions of the doorway and the location of the doorway. Inbuildings under construction, the localization system can act as arecording mechanism for progress and accuracy of the structure and itsfeatures. In another embodiment operator 3510 moves along path 3504 tolocation 3520 within the structure and within view of window 3522 androof opening 3524. In one embodiment coded localization system with FOV3526 measures the roof opening dimensions and location, and FOV 3528measures reference point 3590 and also sky and skyline structure 3594 tofurther localize roof opening 3524. In one embodiment sky structure maybe the magnitude 3582 and angle 3584 of the partial polarization of thesky 3580. By measuring opening 3524 and its relationship and orientationto references external to the structure trajectory planning forconnecting structures, pipelines, transmission line corridors andventilation between distant structures can be achieved with highprecision. In complex building structures where diverse portions of thestructure must meet within millimeters over long distances suchtrajectory estimation and planning can be essential in modeling andcontrolling the building progress and precision.

In another embodiment operator 3510 moves again from location 3520 alongpath 3506 to a location strictly within the interior of the structure3550 and without access to external references. In an embodiment theoperator utilizes coded localization systems with fields of view 3555 toview details 3560 and 3565. Details 3560 and 3565 are for exampleobjects and visual structure within the field of view including textilesand random structure in walls and wall covering, floors and floorcovering, and shadows from lighting discontinuities and highlights fromstructured illumination. In an embodiment, operator 3510 is able to mapin absolute coordinates at least one of the structure and objects withinthe structure without the aid of GPS. In an embodiment, a method andapparatus for performing at least one of distant object orientation and3D location estimation for input to model building process.

In another embodiment FIG. 36 shows the use of coded imaging channels toadd information for human viewers. In many systems image information forhuman viewers is increased by increasing contrast. As the intensity ofthe illumination or length of exposure in these systems decreasescontrast unavoidably decreases, as shown in 3610. The lower theillumination level received from the object the lower the contrast andthe harder for a human viewer to understand, detect or use the image. Inone embodiment coded localization systems can be used to increasecontrast for human viewers by fusing information in other domains thathave contrast that is insensitive to illumination levels compared toclassical imaging systems. In another embodiment coded polarizationsystems have an amplitude that is insensitive to illumination, as shownin graph 1700 of FIG. 17. Dramatically reducing exposure level does notdecrease the amplitude of the partial polarization and results in lowershot noise component of the measured signal. The amplitude component ofpolarization signal related to 3610 is shown in 3620.

In another embodiment of adding information for human viewers the phaseof polarization measurement vector has a contrast, where contrast forphase is defined as the range of −pi to pi, that is insensitive toillumination as long as the amplitude of the partial polarization can bereliably estimated. The polarization phase of the signal related to 3610is shown in 3630.

A great many objects have a natural signal or signature in thepolarization domain. Generally smooth surfaces, man-made surfaces, skin,glass and metal can have distinct polarization signatures that canenable low-cost polarization contrast. In an embodiment if the incidentpolarization is known or can be calculated then the polarization signalcan also be used to estimate the 3D geometry of the object. In anotherembodiment the polarization signal can also be used just as acontrast-enhancing feature and fused with conventional imagery or afunction of the polarization signal (such as an outline of thepolarization signal spatial boundaries) can be fused with signals fromother domains to aid human viewers. In another embodiment polarizationsignals can also be used for monitoring change detection and can also beused with computer processing for automatic detection, recognition andestimation.

In an embodiment FIG. 37 shows the mixing of time, space andpolarization in order to further detection, recognition or estimation ofsignals. Box 3710 shows an object relative to some reference and apolarization measurement P1. Box 3720 shows the same object has movedrelative to the reference and another polarization measurement Pn istaken. In one embodiment a sequence of polarization measurements can betaken, such as from FIG. 3, while the object is in motion and thepolarization signature will contain a component due to spatial motion.Box 3730 depicts a scene with generally low contrast in a wavelengthdomain. By using polarization measurements contrast can be improved asin Box 3740 by using polarization magnitude and angle and also by fusingpartial polarization vector estimates into the wavelength domain. In oneembodiment the phase and amplitude of the polarization signature can beused in a low light situation to add contrast to detection orlocalization. In another embodiment when the polarization signaturerelated to the stationary object is known then the motion of the objectcan be estimated by selecting the spatial points from each measurementimage such that the known polarization stationary signature is formed.In one embodiment, object motion can be compensated or estimated throughknowledge of the stationary polarization signature.

FIG. 38 shows another embodiment of the invention in system 3800 thatdepicts a high resolution imaging system 3810 coupled to a codedlocalization system 3820. In one embodiment the coupling consists ofboresighting systems 3810 and 3820. In one embodiment high resolutionimaging system 3810 forms high resolution image 3830 while codedlocalization system 3822 forms coded vector space 3860 from anarrangement of coded localization sensors 3822 that form system 3820. Inone embodiment coded vector space 3860 is compared via process 3865 tosub-image 3850 extracted via process 3840 from region 3835 in primaryhigh resolution image 3830. In one embodiment process 3865 localizes thefeature set contained within 3860 to the high resolution image 3830. Inanother embodiment feature set 3860 and high resolution image 3830 areused to form a range map of scene contained in image 3830.

FIG. 39 shows an embodiment of a device for a polarization sky compassutilizing the coded designs and processing systems disclosed herein toestimate polarization patterns in the sky when obscured by an unknownmedium as disclosed previously in FIG. 4. System 3900 consists of codedlocalization imaging system 3904 containing a 2×2 arrangement of imagingchannels with overlapping fields of view 3902. Each imaging channelcontains a linear polarizer 3930 oriented at a certain angle to a fixedreference. In this embodiment the angles are selected to form a 2pisymmetric sampling of the unit circle 3908 which enables the dataprocessing disclosed. In this case the angles or samples numbered 1through 4 are shown as linear polarizer angles 3930, correspondinglocations on the unit circle 3908 at 90 degree phase intervals ofrotation (which corresponds to 45 degree interval rotations of thephysical polarizer acting as an analyzer), and resulting images 3920that are formed by system 3904. The processing for the 2×2 system asintroduced in FIG. 3 is shown in graph 3950 where each sample is stackedto form a vector V by process 3975 which collects overlapping pixelsamples from a Q×Q sized window 3922 through the stack of 4 images 3920.In this embodiment, V=[M1(1,1) M2(1,1) M3(1,1) M4(1,1) M1(1,2) M2(1,2) .. . M3(Q,Q) M4(Q,Q)], where only the first few samples are shown inGraph 3950. In these samples the number of the image or data “M”corresponds to the unit circle angle 1 through 4, and the (x,y) locationcorresponds to an image location. To extend graph 3950 to a usableamount of data, a Q×Q window 3922 in each image is selected forcollecting data. At each image location in (x,y) within the Q×Q windowthe pixel intensity at that location is collected for each of the imagesM1 to M4 and arranged in sequence, forming a 4×Q×Q long vector similarto the long vector of samples 1530 in FIG. 15. Q is selected in thisexample to be large (>10) since the clouds forming the obscuring mediumare considered to be very dense and highly attenuating of the skypolarization signal. As shown earlier in FIG. 15, the vector V is thenFourier transformed to form sinusoidal peak locations similar to 1545.Sliding the Q×Q window across the entire image in x,y via process 3970and extracting the angle of the Fourier component from V yields theangle of partial polarization 3990 as derived from the sky images 3920.The magnitude and phase of a particular set of points from the FourierTransform can be also calculated through an efficient sum-of-productsmethod. The point of convergence of the angles in 3990 represents theanti-sun in this example. With the heading of the anti-sun, and knowingthe time of day and geographic location, the device provides a highlyaccurate compass heading angle for use with vehicle control, personalnavigation and other systems requiring accurate heading estimates. Thiscompass does not rely on magnetic fields and can be used on heavymachinery in the presence of large amounts of iron products thatnormally disrupt a magnetic compass.

FIG. 40 shows another embodiment of a device for a polarization skycompass utilizing the coded designs and processing systems disclosedherein to estimate polarization patterns in the sky when obscured by anunknown medium as disclosed previously in FIG. 4. System 4000 has acoded localization imaging system 4004 containing a 3×3 arrangement ofimaging channels with overlapping fields of view 4002. Eight of theimaging channels contain a linear polarizer 4030 oriented at a certainangle to a fixed reference. In this embodiment the angles are selectedto form a 2pi symmetric sampling of the unit circle 4008 which enablesthe data processing disclosed. One channel is uncoded to form a clearimage and is labeled ‘b’ in the image set 4020. In this case the anglesor samples numbered 1 through 8 are shown as polarizer angles 4030,corresponding locations on the unit circle 4008 at 45 degree phaseintervals (which corresponds to 22.5 degree interval rotations of thephysical polarizer), and resulting images 4020 that are formed by system4004. The processing for the 3×3 system as introduced in FIG. 3 is shownin graph 4050 where each sample is stacked to form a vector V bycollecting overlapping pixel samples through the stack of 8 images. Inthis embodiment, V=[M1(1,1) M2(1,1) M3(1,1) M4(1,1) M5(1,1) M6(1,1)M7(1,1) M8(1,1), M1(1,2) M2(1,2) . . . M7(1,2) M8(1,2) M1(1,3) . . .M7(Q,Q) M8(Q,Q)], where only the first few samples are shown in Graph4050. In these samples the number of the image or data “M” correspondsto the unit circle angle 1 through 8, and the (x,y) location correspondsto an image location. To extend graph 4050 to a usable amount of data, aQ×Q window in each image is selected for collecting data. At each imagelocation in (x,y) within the Q×Q window the pixel intensity at thatlocation is collected for each of the images M1 to M8 and arranged insequence, forming a 8×Q×Q long vector similar to vector 1530 in FIG. 15.Q is selected in this example to be large (>10) since the clouds formingthe obscuring medium are considered to be very dense and highlyattenuating of the sky polarization signal. The vector V is then Fouriertransformed to form sinusoidal peak locations as in the previous examplein FIG. 39. This is a similar processing to the system in FIG. 39 exceptthat the 3×3 configuration allows capturing twice as many angles of theprocess as evident in graph 4050. More samples in angle enables higherSNR, or reducing Q for more spatial resolution without sacrificing SNR.Sliding the Q×Q window across the entire image in x,y with process 4070and extracting the angle of the Fourier transform of the data yields theangle of partial polarization 4090. The angle of the partialpolarization is the angle of the complex vector of the Fourier componentdefined by an 8 sample period, as derived from the sky images 4020.Since Q is the same as in FIG. 39 the result 4090 has a higher SNR andless noisy than 3990 in FIG. 39. The point of convergence of the anglesin 4090 represents the anti-sun in this example. With the heading of theanti-sun, and knowing the time of day and geographic location, thedevice provides a highly accurate compass heading angle for use withvehicle control, personal navigation and other uses of headingestimates. This compass does not rely on magnetic fields and can be usedon heavy machinery in the presence of large amounts of iron productsthat normally disrupt a magnetic compass.

FIG. 41 describes a high-speed localization system that has milli-radianestimation precision, small size and low cost. Two systems 4101 and 4102are mounted on a device, such as an arm or boom of a controlled system,where the relative distance and angle of the set of parallel systems isneeded. Each system consists of a two sets of coded high speed angleestimation sensors light meter, and an illumination device, such as anLED. The optics form space-variant measurements by using specializedrelative illumination 4110. The sensors and illumination LEDs canoperate in narrow UV, IR or visible wavelengths ensuring that there areno other sources possible in the field of view of the systems. The fieldof view of the LED and the collection optics 4122 for the codedestimation systems and light meters 4120 is sufficient to cover theexpected relative motion of the parallel systems. This particularexample has a FOV of 60 degrees, or + and −30 degrees. The codedestimation system has a specially designed relativeillumination-vs.-field angle as shown in 4140. The relative illumination4141 varies from 1.0 when the object is on-axis to 0.8 when the objectis at 30 degrees. The estimate of the object angle given the measurementv1 at high speed single pixel detector 4130 measuring v₁ is v₁*K1+K2,where K1 and K2 depend both on the total amount of power reaching theaperture of the estimation system, the efficiency of the detector andthe particular code. By measuring and calibrating the total powerreaching the aperture, with the light meters, the constants K1 and K2can be found. By repeated measurements of the received power, throughthe light meter, any changes in power due to electrical fluctuations,large distance variations, atmospheric conditions, etc. can be accountedfor.

The prescription for the optical system in FIG. 41 is as follows (inZemax Lens units) where CSBR is Cesium Bromide and KCL is PotassiumChloride, both of which can be compression molded:

Surface 1 Surface 2 (stop) Surface 3 Material = CSBR Material = KCLMaterial = Air Radius = 4.001137 Radius = infinity Radius = −0.671541Thickness = 5.88603 Thickness = 2.134825 Thickness = 0.979269 Semi-dia =2.37714 Semi-dia = 0.288709 Semi-dia = 0.99885 Conic = 0.366887 Conic =−5.234306E4 4^(th) = −1.394775E−3 4^(th) = −0.034597 6^(th) =−2.470589E−5 6^(th) = −0.041182 8^(th) = −1.963808E−5 8^(th) = −0.014880

The MTF of this specialized imaging system is very high as shown in4150. The diffraction-limited values are given by 4152 and the on-axisMTF is given by 4151. The off-axis MTFs are similar.

The particular single pixel sensor, LED and optical configuration inFIG. 41 has a combined SNR of 1/500, or one part in 500, for an on-axisobject. The uncertainty of the angle estimate is then given byapproximately [30 degrees/500/0.2 variation] or about 5 milli-radians.Using larger collection optics, a higher power LEDs or more efficientdetectors can increase the angular precision even more. Knowing thebaseline separations D and coordinates of the two corresponding codedestimation systems, from 4103, results in two estimated linear equationsrepresenting the equations to the object 4106 (x₃,y₃) from the two endsof the baseline. Combining the two estimated equations for lines 4104and 4105 via y₃=m₁x₃+b₁=m₂x₃+b₂ results in an estimate of the distantsource (x₃,y₃), as well as the distance and relative angle to the LED.As there are parallel estimation systems, 4102 can also be used tocalculate the relative range and angle to the LED of 4101. Smallvariations of the two estimates due to noise can be averaged outreducing the uncertainty of the final measurements. Any largediscrepancies in estimated range and angles from 4101 and 4102 indicatesomething is wrong with one or the other system or that some externaleffect, like a distance second source, like the sun, is interfering withthe measurements. By watching the relative received power from the lightmeters many of the discrepancies can be understood and corrected.

A step-by-step algorithm is:

-   -   1. Calibrate coded pixel sensor with light meter:        V₁*K1−K2=estimated angle    -   2. Estimate equation of line to LED relative to system baseline        D and coordinate system    -   3. Estimate equation to LED for corresponding estimation system        (y₂)    -   4. Determine orientation of system (x₃,y₃) via        y₃=m₁x₃+b₁=m₂x₃+b₂    -   5. Compare/average orientation estimate with symmetric system.        Exploit symmetry.

FIG. 42 shows a preferred embodiment of a device for imaging using thecoded designs and processing systems disclosed herein. System 4200consists of coded localization imaging system 4204 containing a 2×2arrangement of imaging channels with overlapping fields of view 4202 andimaging model M=C*I where M is a matrix of sampled data, C is a matrixthat describes the code space, and I is a matrix of the object intensityvalues, and where * denotes matrix multiplication.

Each imaging channel contains a pixel-level grayscale intensity code4230 based on a biased sinusoid and cosine with a phase shift of 135degrees and orientation of 0 and 90 degrees. For w=pi*(2/pixel size) andf=135 deg, the codes C1 to C4 are for channels 1 to 4 are:

C1=½ sin(w*x+f)+½

C2=½ cos(w*y+f)+½

C3=C2^(T)

C4=C1^(T)

Where the period is chosen so that with the independent variables x andy one period of sinusoidal codes occupies one pixel width. Theindependent variable extends to all pixels in the particular channels.When the independent variables x and y are chosen so that they areinteger multiples of +(pixel size/2) and −(pixel size/2) each pixel is emodified by two grayscale values as shown in 4231, 4232, 4233, and 4234where the phase and amplitude for a single pixel for each code is shown.For example, 4231 shows a close view of the code pattern over a singlepixel where the right half of the pixel is clear and the left half ofthe pixel has an attenuating grayscale value over it. In this examplethe angles are selected to form a symmetric sampling of the phase andamplitude of spatial frequency space 4208 which enables the dataprocessing disclosed. The samples numbered 1 through 4 are shown as graycodes 4230, corresponding locations in frequency space 4208, andresulting images 4220 that are formed by system 4204. Frequency space4208 shows that the complex sampling achieved in the coded system is 2×what the pixel spatial frequency allows. The unmodified pixel allows thecentral square labeled “b” in 4208, while the coded pixels push themeasured spectral response out by a factor of 2 by using 2 intensityvalues per pixel. The spectral response is pushed out further to 3× thepixel sampling frequency in the next example FIG. 43. The response ofthe central square “b” is not sampled in this case, but is achieved byaveraging all of the 4 coded channels together. In contrast in FIG. 43the actual bias value or native pixel resolution is sampled using aclear channel.

The coded pixels of system 4200 essentially sample the complex amplitudeat spatial frequencies higher than the maximum spatial frequency definedby physical pixels. While any one of corresponding coded pixel pairs4231/4234 or 4233/4232 can yield an estimate of the magnitude of thespatial frequency regions defined by 4208, the pairs of codes are neededto determine both the magnitude and phase of the spatial frequencyregions. In an embodiment, coded localization systems providemeasurement vectors that represent the complex Fourier spatial frequencycomponents beyond the spatial frequency limits of the detector pixels.

The processing for the 2×2 system in FIG. 42 as introduced in FIG. 6 andFIG. 23 is shown as forming a vector V by collecting adjacent pixelsamples through the stack of 4 coded images. In this example, process4270 takes 4×Q×Q samples from each Q×Q window, with 2 grayscale values(or codes) per pixel Q=2, and the matrix V is:

$\underset{\_}{V} = \begin{bmatrix}{M\; 1\left( {1,1} \right)\mspace{14mu} M\; 1\left( {2,1} \right)\mspace{14mu} M\; 1\left( {1,2} \right)\mspace{14mu} M\; 1\left( {1,4} \right)} \\{M\; 2\left( {1,1} \right)\mspace{14mu} M\; 2\left( {2,1} \right)\mspace{14mu} M\; 2\left( {1,2} \right)\mspace{14mu} M\; 2\left( {1,4} \right)} \\{M\; 3\left( {1,1} \right)\mspace{14mu} M\; 3\left( {2,1} \right)\mspace{14mu} M\; 3\left( {1,2} \right)\mspace{14mu} M\; 3\; \left( {1,4} \right)} \\{M\; 4\left( {1,1} \right)\mspace{14mu} M\; 4\left( {2,1} \right)\mspace{14mu} M\; 4\left( {1,2} \right)\mspace{14mu} M\; 4\left( {1,4} \right)}\end{bmatrix}$

In these samples the number of the image or data “M” corresponds to thefrequency domain regions 1 through 4 and the number of channels, and the(x,y) location corresponds to a coded image location. One importantproperty of the set of codes 4230 is that the sum of all the 2D codes isa constant. The sum of the code measurements can then be used as directestimate of the baseband spatial frequency value denoted by “b” in 4208.As described in 4280, an estimate of the bias or baseband pixel value isdetermined by a weighted sum of the coded pixel values. This estimatedbias is then used to remove the bias from the measured data matrix M. Byremoving the bias from the measurement the code matrix (C−½) can bebipolar and composed ideally of orthogonal sinusoids. Without the factorof ½, or bias, the code matrix the components are orthogonal sinusoidswith an ideal inverse that has the minimum effect of noise. In effect,the ideal code matrix (C−½) is a complex Fourier Transform and theinverse matrix is the complex inverse Fourier Transform of a linearcombination of the measured data.

The vector V is transformed in 4280 using a model of the inverse of thecodes (C−½) to form an estimate of pixels in the full resolution imagewith reconstruction model: I_(est)=(C−½)⁻¹ (V−b/2)+b/9 where the value bis given by sum_(i)(Mi)/2. Ideally (C−½)⁻¹ is the 2D inverse FourierTransform matrix.

Sliding the Q×Q window across the entire image in x,y and extracting theintensity estimate by multiplying (C−½)⁻¹ with a scaled version of thedata matrix V plus the bias yields the full resolution image 4290. Thisenables angle coded systems that were designed for localization tasks toalso provide full resolution imagery.

Furthermore, in a coded localization system 4200 viewing a single pointsource object within the field of view, each pixel can estimate theangle to the object, as demonstrated earlier in FIG. 41, due to theintensity coding. This provides the system in FIG. 42 (with many pixels)the capability to estimate range with sub-pixel accuracy to pointobjects as well as form the final image. As described in FIG. 41,multiple coded pixels with a known baseline can perform rangeestimation, hence the system in FIG. 42 with a multitude of pixels andcodes can form range estimates as in FIG. 41 as well as form fullresolution images as in FIG. 42. Ranging to point or other known sourcessuch as retro-reflectors or active sources such as LEDs, andsimultaneously imaging the surrounding environment with preciseregistration to the point source is critical for surveying, mapping, andlocalizing remote objects in targeting applications. FIG. 43 shows apreferred embodiment of a device for imaging using the coded designs andprocessing systems disclosed herein. System 4300 consists of codedlocalization imaging system 4304 containing a 3×3 arrangement of imagingchannels with overlapping fields of view 4302. Eight of the nine imagingchannels contain a pixel-level grayscale intensity code 4330 based on abiased sinusoid and cosine with a phase shift of pi/12 radians andorientations of 0, 45, 90, and 135 degrees and one channel is clear. Inthis embodiment the codes are selected to form a symmetric sampling ofthe complex frequency space 4308 which enables the data processingdisclosed. In this case the angles or samples numbered 1 through 9 areshown as gray codes 4330, corresponding locations in complex frequencyspace 4308, and resulting images 4320 that are formed by system 4304.Each pixel is modified by three grayscale values as shown in 4334 and4337 where the phase and amplitude for a single pixel for each code isshown. For example, 4334 shows a close view of the code pattern over asingle pixel where the top ⅓ of the pixel is heavily attenuated, themiddle ⅓ is the least attenuated, and the bottom ⅓ is attenuated at yeta third grayscale value. Pixel code 4337 also shows only three intensityvalues, this time arranged in a diagonal structure to fill out thefrequency space 4308 diagonal frequency content as indicated by squares1, 3, 7 and 9 which correspond to the diagonal sampling codes in 4330.For w=pi (2/pixel size) and f=pi/12, the codes C1 to C9 for channels 1to 9 are:

C1=½ sin(w*x+f)+½, at −45 degrees rotation

C2=½ sin(w*x+f)+½, at 0 degrees rotation

C3=½ sin(w*x+f)+½, at 45 degrees rotation

C4=½ sin(w*x+f)+½, at 90 degrees rotation

C5=1

C6=½ cos(w*x+f)+½, at 0 degrees rotation

C7=½ cos(w*x+f)+½, at −45 degrees rotation

C8=½ cos(w*x+f)+½, at 90 degrees rotation

C9=½ cos(w*x+f)+½, at 45 degrees rotation

Where the independent variable x is chosen to be three samples across apixel and extend to all pixels in each channel. The period of thesinusoids is equal to the dimension of one pixel in this example. Theperiod of the sinusoids can be integer multiples of the pixel period orcan correspond to any other higher or lower spatial frequency that issupported by the imaging optics and code fabrication. The processing4370 for the 3×3 system is similar to that in FIG. 42 and is shown asforming a vector V by collecting adjacent pixel samples through thestack of 9 images 4320 in Q×Q window 4322. In this embodiment, Q=3 and:

$\underset{\_}{V} = \begin{bmatrix}{M\; 1\left( {1,1} \right)\mspace{14mu} M\; 1\left( {2,1} \right)\mspace{14mu} M\; 1\left( {3,1} \right)\mspace{20mu} \ldots \mspace{25mu} M\; 1\left( {1,3} \right)\mspace{14mu} M\; 1\left( {2,3} \right)\mspace{14mu} M\; 1\; \left( {3,3} \right)} \\{M\; 2\left( {1,1} \right)\mspace{14mu} M\; 2\left( {2,1} \right)\mspace{14mu} M\; 1\left( {3,1} \right)\mspace{20mu} \ldots \mspace{25mu} M\; 2\left( {1,3} \right)\mspace{14mu} M\; 2\left( {2,3} \right)\mspace{14mu} M\; 2\; \left( {3,3} \right)} \\{\text{:}\mspace{410mu}} \\{M\; 8\left( {1,1} \right)\mspace{14mu} M\; 8\left( {2,1} \right)\mspace{14mu} M\; 8\left( {3,1} \right)\mspace{20mu} \ldots \mspace{25mu} M\; 8\left( {1,3} \right)\mspace{14mu} M\; 8\left( {2,3} \right)\mspace{14mu} M\; 8\; \left( {3,3} \right)} \\{M\; 9\left( {1,1} \right)\mspace{14mu} M\; 9\left( {2,1} \right)\mspace{14mu} M\; 9\left( {3,1} \right)\mspace{20mu} \ldots \mspace{25mu} M\; 9\left( {1,3} \right)\mspace{14mu} M\; 9\left( {2,3} \right)\mspace{14mu} M\; 9\; \left( {3,3} \right)}\end{bmatrix}$

In these samples the number of the image or data “M” corresponds to thefrequency domain regions 1 through 9 and the number of channels, and the(x,y) location corresponds to an image location. While any one of thecorresponding codes channels, such as 2/5, 3/9, 1/7 can be used todirectly sample the magnitude of the spatial frequency regions in 4308,the set is needed to sample and estimate both the magnitude and phase ofthe specific spatial frequency regions. The spatial frequency regionscan be adjusted to essentially spatial frequency that the optical systemand code fabrication will allow.

An increased resolution image is formed by first estimating the bias, orbaseband component for each pixel. This can be done through the valuesof the clear channel, denoted as channel 5. The 8 codes can also bedesigned so that their sum is a constant in 2D, such as in FIG. 42. Boththe clear channel values and the estimated bias values from the sum ofthe codes can be combined to reduce the effects of noise. Themeasurement vector V is then scaled and transformed using a model of theinverse of the codes (C−½), plus a scale value of the bias, to form anestimate of pixels in the full resolution image. Sliding the Q×Q window4322 across the entire image in x,y and extracting the intensityestimate by multiplying (C−½)⁻¹ with (V−b/2), and adding b/9, yields thefull resolution image 4390. Ideally the inverse of (C−½) is the inverse2D Fourier Transform matrix, which has the minimum effect onmeasurements in noise. This enables angle coded systems that weredesigned for localization tasks to also provide high quality fullresolution imagery. Result 4390 also contains a point source of interest4392. Ranging to this point source of interest is discussed in detail inFIG. 44.

FIG. 44 is an embodiment of the invention from FIG. 43 disclosingranging using groups of single pixels. Single pixel range estimationusing coding has also been disclosed earlier in FIGS. 18 and 41 and isdiscussed here in more detail. System 4400 contains 4410 which is acollection of objects including a known source (LED or retro-reflector)to be localized in the field of view. System 4304 from FIG. 43 is usedas the device. The grayscale intensity code sections of rays from thesampled pixels capturing the retro-reflector pass through are shown as4420. The particular region of the code the rays pass through determinethe attenuation of the object intensity value as detected on thedetector and are shown as small circles 4430. Small region of sampleddata M1(x,y) to M9(x,y) centered on retro-reflector is also shown as4440. In 4440 each channel produces a region of pixels surrounding theretro-reflector, numbered 1 through 9 with small circles. The intensityvalue of the pixels that are centered on the retro-reflector are alsogiven in 4440 and, for channels 1 through 9, they are 101, 96, 77, 65,177, 77, 38, 64, and 134 respectively, with an arrow indicating thecentral pixel. Using the detected intensity of any channel inconjunction with the object intensity as measured by channel 5 (theclear channel) allows one to trace a ray 4425 from the central sample in4440 through the appropriate gray code region in 4420 at an angle thatindicates the direction of the point source. The intersection of all therays 4425 represents the range and x,y location of the point source.

In FIG. 44 the location of the object codes relative to the pixels andoptics, as well as the approximate distance to the objects areessentially known. In many cases some or all of these parameters may notbe initially known, such as with systems with fabrication errors ormechanical movement due to temperature or vibration effects. The rangeof the object could also vary. Control of the non-ideal effects due toparameters of the system may be termed calibration. Control of thenon-ideal effects due to parameters of the object, like distance, may betermed focusing. The values and orientation of the code matrix C arethen adjusted to correspond to the measurements V.

It is likely that the matrix condition number of an ideal code matrix C,or (C−½) will decrease due to the non-ideal fabrication and assemblyeffects thereby decreasing the degree of orthogonality of the imagingchannels. If the uncertainty of the fabrication and assembly errors thatinfluence the code matrix C are known in advance then the code matrixcan be purposely designed to maximize the minimum expected singularvalue over the range of errors thereby making the final system 4500relatively insensitive to non-ideal effects from fabrication andassembly. This design can be done by optimizing the code so that afterthe range of expected errors due to fabrication and assembly, such asthrough Monte-Carlo statistical trials, the final code matrix aftercalibration has the maximum minimum singular value and thus the maximumdegree of orthogonality.

FIG. 45 illustrates focusing and codes as a function of range. The chiefrays 4510 and 4511 are from an object whose relative distance isessentially infinity compared to the dimensions of the lens array 4501.The chief rays 4520 and 4521 are related to a relatively close source.The measured values from sensor 4504 change between the distant andclose objects as seen by the change in the location of the chief rays4531/4532 and 4533/4534 at the image planes and through the codes 4502and 4503. The codes 4502 and 4503 spatially change as a function ofposition, similarly to the codes of 4420 of FIG. 44. The array of 3lenses/codes/sensors generally represent the middle row of the system4400 in FIG. 44 where the central channel has no code. As the objectrange changes the image formation model, given by the code matrix C,deterministically changes to reflect the actual formed samples. Ineffect the code matrix C becomes C(range), or the code matrix is afunction of the particular object range and the rows and columns ofC(range) are shifted to account for the deterministic change in objectrange of objects at particular spatial locations. In classical imagingsystems the distance between the lens and image plane is mechanicallymoved to change focus. In coded multi-aperture systems electricallyvarying the code matrix as a function of range, C(range), and thenreconstructed the image, as from FIG. 43, effects a change in focus withno physically moving parts.

If the object range is known, and the calibrated code matrix C known,then the proper image formation matrix C(range) can be formed and thecorresponding high resolution image or object localization can beperformed. If the object range is not known then a series of rangevalues can be used to determine the “best focused” image through thepseudo inverse of (C(range)−½) acting on the scaled measurements V. Bycomparing the maximum spatial frequency content of the formed images ormaximum MTF, for example, an estimate of object range and a clear imagecan be formed. When the spatial frequency content is a maximum theobject can be considered well focused. This is similar to the processingthat is done when mechanically moving the lens in a classical imagingsystem in order to form auto focus. Range estimation and therefore focuscan be performed for every pixel in the image and for every type ofobject. When the object has a broad range of objects and best-focus isfound over all relevant regions in the final images then the codedmulti-aperture system forms an extended depth of focus image. Focusingcan also be done for a small region of the image and the same estimatedrange value used for the entire image. By coding each channel as afunction of range a focusing can be achieved for objects not atinfinity. By coding each channel as a function of range an extendeddepth of field image can be achieved. By coding each channel as afunction of range an estimate of range for points in the image can beachieved.

FIG. 46 is an embodiment of a navigation system for dead reckoning.System 4600 depicts an automated rolling robot cart 4610 that movesalong a service pathway 4606 that goes through tunnel 4604 beneathroadway 4602. During operation outside the tunnel the cart navigates byusing GPS 4615 which provides location information and a polarizationsky compass 4618 that provides compass heading information. The knownsky polarization pattern exhibits symmetry so in this example thepolarization compass only views a portion of the sky and uses rules ofpattern symmetry to determine the heading angle. The cart is made ofiron and moves iron material along the pathway so a magnetic compass isvery inaccurate. The polarization sky compass 4618 provides headinginformation as long as a portion of the sky is present. Inside thetunnel neither GPS nor a view of the sky is available for navigation.Inertial navigation is known and well understood to be failure prone dueto drift. Inertial navigation systems include for example accelerometersand rate gyroscopes where the signals are integrated to produce adisplacement estimate in x, y, z and roll, pitch and yaw. To alleviatethe drift and inaccuracies of inertial systems for dead reckoning withinthe tunnel, the system 4600 operates using optical motion units 4620,4622 and 4625 which are configured to determine motion in the field ofview as disclosed initially in FIG. 9. In this case the basic conceptintroduced in FIG. 9 is configured to provide navigation, and is alsocoupled with other systems disclosed herein to provide numerous benefitsover the prior art.

The motion detected by a single system like 4625, containing optics4662, detector 4664 and optical flow processing engine 4666 is anymotion that causes the image to shift and can be detected by the opticalflow processor 4666. The number of detector elements in 4664 ispreferably small for example 32×32 to enable efficient optical flowprocessing. Pixels may also be large to enable efficient light captureas image resolution is not often the final goal. A single lens with a 30degree field of view focused to infinity, a 32×32 detector and acorrelation engine based on a digital signal processor (DSP) thatspatially correlates images from two points in time to determine theshift in global features is a simple implementation for 4625. Memoryonboard the DSP is used to store the images. Another implementation of4666 is using a classic gradient optical flow algorithm based on anFPGA. Since the gradient algorithm is based on the assumption that apoint of light does not change intensity as it moves using a codedintensity mask on the detector is not an option for this algorithm. Asdescribed in FIG. 9 one embodiment of the invention is to arrangeseveral optical flow engine devices around a rigid body, in this casethe rigid body is the automated cart 4510. The rigid framework of thecart couples the motion detected in the disparate units 4620, 4622, and4625 to a motion model for translation and rotation of the cart. Forexample 4630 is a top view of the system 4600 with cart 4610. Axes areshown as Y and X in the plane of the drawing. When the cart movesforward in the middle of the tunnel the optical motion units 4620 andthe pair 4625 will all observe the same dx value. If the cart is notgoing straight forward but is for example drifting to one side the dxvalues observed will increase on one side of the cart and decrease onthe other side of the cart. For example samples indicating a straightforward motion may look like: dx₄₅₂₀=[1 2 2 3 3] and dx₄₅₂₅: [1 2 2 33], i.e. the steps in dx are identical, and in this example the cart isspeeding up since the dx values are increasing. An increasing drifttoward 4620 may look like: dx₄₅₂₀=[1 1 2 2 3] and dx₄₅₂₅: [1 1 2 1 1],with the drift beginning around the 4^(th) sample as seen in thedifference of the two signals, i.e. dx₄₅₂₀−dx₄₅₂₅=[0 0 0 1 2]. Thecart's control system can now take corrective action to steer and avoidthe drift. If the cart remains off center then the difference willremain non zero but will be constant for a straight forward trajectory.The cart's control system may also choose to re-center itself by drivingdx₄₅₂₀−dx₄₅₂₅ back to zero in this case. The optical motion unitprovides dead reckoning for navigation.

Vertical motion is indicated by dz in 4620 and 4625 according to theside view 4640. Pitch, roll and yaw can also be measured. A change indifference of dz values between 4620 and 4625 indicates a rollingaction. Pitching in this example is indicated by a non-zero differencein dz values from the pair of sensors 4625. Yawing is determined by adifference in dx values for the pair of sensors 4625, although weaklyso. Yawing is also measured with more precision by using the sensor 4622on the tail of the cart. 4622 is shown in side view 4640 and beingangled down to view the surface 4606. The sensor 4622 will measureforward motion as dx, and lateral motion as dy. Motion in Z will alsoaffect 4622 depending on the angle viewing the ground, and height can bedetermined in this case by dz measurements.

Since the optical motion units are based on imaging systems they canalso capture simple images and document the tunnel in this application.Images captured may be coded images as previously described to enablesub-pixel ranging. Details about the surfaces and algorithm results mayalso be provided to the cart as previously described in FIG. 9.Knowledge about SNR from the details can be used in the guidancealgorithm to reject sensors that become dirty or do not see sufficienttexture to perform optical flow calculations in this example.

FIG. 47 is an embodiment of a polarization compass. Arrangement 4701 isa temporal sampling polarization compass consisting of a commerciallyavailable CMOS industrial USB3-connected camera 4710 with nominal bodydimensions A=35 mm, B=35 mm. Also with industry standard “C/CS” lensmount bezel 4711 and industry standard ¼″-20TPI threaded mounting hole4713 in the center of the camera body. The sensor in camera 4710 is a1.3MPixel On Semi VITA1300 CMOS, ½″ sensor format, 4.8 μm pixel sizewith global shutter generating 1280×1024 pixels at a maximum of 150frames/sec. Lens 4715 with nominal dimensions C=34 mm, D=32 mm andindustry standard “C/CS” lens mount thread 4714 screws into industrystandard “C/CS” lens mount bezel 4711. Lens 4715 may be a SchneiderOptics Cinegon f/1.4, focal length 8 mm compact C-mount lens with anominal field of view K=60 degrees and a front thread 4716 to acceptmetric thread M30.5×0.5 filters 4719 and 4719 a that have M30.5×0.5mating thread 4718. Filter 4719 in this arrangement is a linearpolarizer with 95% efficiency, 30% transmission and nominal dimensionsof E=32 mm, F=10 mm, for example NT46-574 from Edmund Optics. Filter4719 a is a blue band pass filter, for example a Hoya B-440 orequivalent with a pass band between 395 nm to 480 nm and nominaldimensions of G=32 mm, H=5 mm, for example NT46-547 from Edmund Optics,that mounts to filter 4719 with lens mount thread 4718 a. The bluefilter 4719 a blocks red and green light which are not well polarizedfrom striking the polarizer filter 4719. Linear polarizer 4719 is on theoutside of the lens since it is easy to mount there with the M30.5×0.5thread, and also because polarizers reflect or absorb light and if thepolarizer were between the lens 4715 and the camera 4710 it may inducestray light into the imaging path or heat into the sensor both of whichwill degrade the signal quality. Also each field of view may not be ableto image the full sky due to obstructions such as 4772 in scenario 4770.In this case the sky polarization pattern that is visible 4771 is knownto have at least one axis of symmetry. In this case the dotted lines in4775 represent the polarization pattern reconstructed for the full fieldof view using rules of symmetry about a horizontal line. This enablesthe device to form full polarization patterns of the sky in the presenceof obstructions, and also determine heading angle using knowledge of skypolarization symmetry without requiring a fully unobstructed view.

Camera 4710 is powered via the USB3 cable 4750 which is plugged intolaptop 4780. Laptop 4780 is for example powered by battery 4799 and isusing an Intel Core i5 CPU at 2.5 GHz with 4 GB of RAM and a 32-bitoperating system. Laptop 4780 is running algorithm consisting of signalprocessing blocks disclosed earlier. The algorithm first samples data in4781 by interacting with the user to rotate the polarizer 4718 whiletaking pictures with camera 4710 and lens 4715 with filters 4719 and4719 a. The polarization angles the users selects are sampled around theunit circle in a uniform fashion as disclosed earlier. The user isencouraged to sample as quickly as possible to avoid temporal issuesfrom moving clouds, trees, birds and airplanes for example. Camera 4710should also be mounted to a tripod or other fixed structure usingmounting hole 4713 to avoid camera motion during subsequent exposures.The algorithm next sorts the data in 4782 as disclosed earlier forpolarization processing, collecting samples across k for Mk(x,y) imagevalues. Example code in 4782 for stacking the data in a Q×Q window with8 polarization samples into a Q×Q×8 length vector V is:

for y=1:Q, for x=1:Q, for a=1:8, V(k)=M(y,x,a); k=k+1; end, end, end

where the third dimension in M is the image value, and is indexedsimilar to how Matlab software naturally arranges 3D matrices. The Q×Qregion is panned across the image set in x,y to form a full field ofview polarization angle and magnitude result. The algorithm then Fouriertransforms the vector V in 4783 by using a standard FFT (fast Fouriertransform) technique, producing Q*Q complex values representing thespectral content of the vector V. The spectral content, call it S, isarranged in a vector with the DC value first followed by the positivefrequency coefficients with increasing frequency index, and then thenegative frequency coefficients. The coefficient of interest is in thefirst half of S so the algorithm in 4784 then selects the frequency dataof interest from S by taking the magnitude and phase angle of thecoefficient at index Q*Q*8/8+1, or S(Q*Q*8/8+1), where the ⅛ is due tothe user taking 8 images in the sampling portion 4781. The algorithmthen displays the magnitude and angle for this Q×Q region in 4785 andthe Q×Q region is panned across the image set in x,y to form a fullfield of view polarization angle and magnitude result.

Arrangement 4702 is a spatially sampling polarization compass and sharesmany of the features of the arrangement 4701 by using four cameras andlenses and filters in a 2×2 arrangement (only two are shown as 4702presents a side view, the other two cameras and lenses and filters arebehind the ones shown). Camera 4720 and 4730 are identical to 4710, withlenses 4725 and 4735 identical to 4715, and filter set 4729 and 4739identical to 4719+4719 a except that the linear polarizers 4729 and 4739are at 45 degrees to each other. The four cameras therefore sample at 0deg, 45 deg, 90 deg, 135 deg and images will be taken simultaneously bythe algorithm described below. In this arrangement the cameras 4720 and4730 are also mounted to a common rigid plate 4740 with ¼″-20TPImounting studs 4741 and 4742 that mate to mounting holes 4723 and 4733.Plate 4740 is for example 60 mm×60 mm×10 mm thick aluminum plate.Cameras 4720 and 4730 are powered by USB3 cables 4760 from laptopbattery 4799 and also send data back to the computer over cable 4760.The two other cameras and cables are not shown. In this arrangement thealgorithm in laptop 4790 (this laptop is identical to laptop 4780) isbased on spatial sampling and not temporal sampling. The algorithm firstsamples data in 4791 by interacting with the user to take a simultaneousimage using all four cameras. Fields of view J are overlapping and ofsimilar dimensions to field of view K described earlier. By takingsimultaneous images the temporal sampling problems described earlier(motion of objects and the sampling rig) are largely alleviated at leastto the extent of the shutter exposure time for the system. The remainingportions of the algorithm 4792, 4793, 4794 and 4795 are similar to thosedisclosed for the temporal sampling system, 4782, 4783, 4784 and 47895,respectively. In this case with 4 samples, example code in 4792 forstacking the data in a Q×Q window with 4 polarization samples into aQ×Q×4 length vector {grave over (V)} is:

for y=1:Q, for x=1:Q, for a=1:4, V(k)=M(y,x,a); k=k+1; end, end, end

In 4793 the spectral content S of the vector V is again formed. Thealgorithm in 4794 selects the frequency data of interest from S bytaking the magnitude and phase angle of the coefficient at indexQ*Q*4/4+1, or S(Q*Q*4/4+1), where the ¼ is due to the arrangement 4702producing 4 images in the sampling portion 4781. Those skilled in theart can see that a temporal sampling arrangement 4701 is easily extendedto a spatial sampling arrangement 4702, and that 4702 can be easilyextended to include 8 camera+lens+filter subsystems, where polarizerangles would be optimally arranged at 22.5 degrees instead of 45degrees, providing a finer sampling around the unit circle as comparedto 4 images.

In another embodiment the cameras 4710, 4720 and 4730 contain built-inprocessing such as industrial “smart” cameras. Such cameras provide anon-board DSP or FPGA that can perform some basic pre-processing. Apre-processing step that is useful to perform outside of the laptop 4780or 4790 using such on-board processing is to bin the pixels from thecamera prior to transmission to the computer. The binning operationrecommended is the summation of neighboring pixels to produce a smallerimage where each binned result contains the A/D counts from the adjacentpixels. This summation process averages out the random noise andprovides a higher SNR in polarization signal, at the expense of spatialresolution. Since the polarization signal varies slowly across the skythis is a useful technique for extracting low SNR signals such as arecaused by clouds and trees, while alleviating the laptop from the burdenof receiving and binning pixels. This distribution of processing betweenan on-board camera processor and the downstream laptop allows the laptopto be replaced by a simple DSP or FPGA and external battery supply.Those skilled in the art can recognize that a laptop performing analgorithm as simple as sorting data in Mk(x,y) and calculating a Fouriertransform can be easily replaced with an FPGA or DSP. Those skilled inthe art also recognize that since a single spectral coefficient isextracted from the Fourier data, that the FFT algorithm can also bereplaced with an inner product of the sinusoid of interest, and willalso recognize this as a narrow band pass filtering operation similar toextracting a single sinusoid from a spectral estimate.

FIG. 48 is an embodiment of an optical motion and motility unit with 5arms as disclosed in previous figures and consists of two circuit boards4801 and 4851 that connect at 90 degree angle via connector 4810.Primary circuit board 4801 has dimensions H1=68 mm, W1=118 mm Extensioncircuit board 4851 has dimensions H2=68 mm, W2=60 mm Dark circles showlocations of optics 4821, 4822, 4823, 4824 and 4825 on each circuitboard. The optics are industry standard M12 lenses in an aluminum mountthat is epoxied to the underside of the circuit board. The primarycircuit board optics 4821, 4822 and 4823 have focal lengths 2.2 mm toproduce field of view A. The extension board has optics 4824 4825 with 8mm focal length to produce field of view B. Primary circuit board alsocontains a micro-controller 4860 based on an 8-bit Atmel ATmega 328micro controller operating at 3.3V and 8 MHz. The controller 4860collects dx, dy information from sensors beneath the lenses. Sensormounting pads are evident on the circuit board layouts. Each sensorbehind the lenses is an Avago ADNS-3080 sensor with 30×30 image arrayand optical flow engine. The sensors are positioned at 120 degrees toeach other on the primary circuit board, and 180 degrees from each otheron the extension circuit board. This placement is used to arrange theoptical axes as close together as possible on the primary circuit board,with 120 degree rotation providing a full sampling of orthogonaldirections for dx, dy sampling disparity, while also providing thewidest separation of imaging systems on the extension board for maximumspatial disparity of the optical axes. This enhances differentiation ofmoving objects passing by the sensor pair, for example while performingrange profiling as demonstrated in FIG. 32, while still allowingelectrical routing on a simple double-sided FR4 circuit board. Theoptical mounts are epoxied to one side of the circuit board and thelenses provide aerial images through a hole in the circuit board. Thesensors for each lens are mounted on the opposite side of the circuitboard as the lenses with the sensor facing the hole, and view the aerialimage through the hole in the circuit board. The side view and top viewin graphic 4891 further illustrate the locations of the components forthe assembled system. In this configuration the system operates as anoptical motion and motility unit with 5 arms, with three sensors lookingdownward and two sensors looking outward, as previously disclosed inFIG. 21. With the physical construction of FIG. 48 it can be seen thatbuilding an optical motility and motion detection system is fairlystraightforward now that the concept has been fully described. Thissystem is powered by 3 AAA batteries connected in series, or 4.5V, andprovides 3.3V at 500 mA to the controller and all sensors using avoltage regulator on the primary circuit board. This system alsotransmits via serial port from the micro dx, dy, dz raw data, processeddata, and raw or processed image data. Processing for example is thedecomposition of the dx, dy, dz signals from individual sensors into aglobal motion for the rigid body (the rigid body in this case is theprimary circuit board+extension board) following the type ofdecomposition disclosed previously in FIG. 9 and provided in the ATMegacode example below based on the Arduino programming language. In thiscode example the globals gx1, gy1, gx2, gy2, gx3, gy3 are theaccumulated global x,y location of each of the optics 4821, 4822, 4823respectively, where the accumulation is the accumulation of decomposeddx and dy motions over time. Furthermore, the decomposition of thevectors depends on the present global angle ‘gT’ in the code below, sothis angle is also updated at each cycle.

#define r30 (30 * PI / 180) // 30 deg in radians #define r60 (60 * PI /180) // 60 deg in radians gx1 += (float)dx1 * cos(gT) − (float)dy1 *sin(gT); gy1 += (float)dx1 * sin(gT) + (float)dy1 * cos(gT); gx2 +=−(float)dx2 * cos(r60 + gT) + (float)dy2 * cos(r30 − gT); gy2 +=−(float)dx2 * sin(r60 + gT) − (float)dy2 * sin(r30 − gT); gx3 +=−(float)dx3 * cos(r60 − gT) − (float)dy3 * cos(r30 + gT); gy3 +=(float)dx3 * sin(r60 − gT) − (float)dy3 * sin(r30 + gT); // update theglobal angle float theta01 = atan2(gy2 − gy1, gx2 − gx1) + 60 * PI /180; float theta02 = atan2(gy3 − gy1, gx3 − gx1) + 120 * PI / 180; floattheta12 = atan2(gy3 − gy2, gx3 − gx2) − 180 * PI / 180; // averageangles using real/imag to avoid wrapping problems float re =cos(theta01) + cos(theta02) + cos(theta12); float im = sin(theta01) +sin(theta02) + sin(theta12); gT = atan2(im, re);

The global motion for the other two optics 4824 and 4825 follow asimilar construction, except that the global angle constants are 0 and180 deg, not 60, 120, and 180 as shown in the code above, and theextension board will use a different global theta rather than ‘gT’, callit ‘xT’ for extension board. If the primary circuit board 4801 is heldhorizontal to the ground and the extension circuit board 4851 is facingforward or rearward in a moving vehicle, then ‘gT’ represents yaw angleand ‘xT’ represents roll angle when the sensors are viewing the worldoutside the vehicle. If the primary circuit board 4801 is heldhorizontal to the ground and the extension circuit board 4851 is facingto the side in a moving vehicle, then ‘gT’ again represents yaw angleand ‘xT’ represents pitch angle

FIG. 49 is an embodiment of the invention showing details on the imagingsystems previously described in 3904 FIG. 39, 4004 FIG. 40, 4204 FIGS.42, and 4304 FIG. 43. Lens layout 4901 with detector 4903 is for asingle channel of the 2×2 and 3×3 imaging systems shown earlier. Thematerial 4902 is known as Zeonex 480R. The material 4902 b is ZeonexE48R. The prescription for the optical system in 4901 is as follows (inZemax Lens units):

Surface 1 Surface 2 Surface 3 (stop) Material = Polystyr Radius =6.268815 Radius = infinity Radius = 21.112468 Thickness = 1.980373Thickness = 0.362479 Thickness = 3.506021 Semi-dia = 2.468438 Semi-dia =2.331975 Semi-dia = 3.08 Conic = −1.004726 Conic = −36.22703 4^(th) =−2.175187E−3 4^(th) = −1.154485E−3 6^(th) = 3.003577E−5 6^(th) =−2.186555E−5 8^(th) = 1.582201E−6 8^(th) = 8.787575E−7 Surface 4 (Binary2) Surface 5 Surface 6 Material = 480R Radius = −11.519072 Material =E48R Radius = 14.390332 Thickness = 0.199356 Radius = 5.271419 Thickness= 2.0001 Semi-dia = 2.856551 Thickness = 3.144312 Semi-dia = 2.589251Conic = 0 Semi-dia = 3.100682 Conic = 0 Conic = 0 Diffraction Order = 1Surface 7 Surface 8 Surface 9 Radius = −11.137381 Material = PolystyrRadius = 4.179707 Thickness = 0.388846 Radius = −30.83006 Thickness =5.859802 Semi-dia = 2.834143 Thickness = 1.999658 Semi-dia = 2.303156Conic = 0 Semi-dia = 2.63680 Conic = −3.473208 Conic = 43.8999648 4^(th)= 2.954178E−3 4^(th) = −3.868053E−3 6^(th) = −2.756575E−5 6^(th)=1.012411E−4 8^(th) = 1.218266E−5 8^(th) = 4.367926E−7

The MTF of this specialized imaging system is very high as shown ingraph 4905. To produce the coded localization polarization compassdisclosed herein, a linear polarizer is added to the front of eachchannel as in 4920. To produce a coded localization system with agrayscale pixel-scale code, the intensity code for each channel isplaced on top of the detector array as in 4925. This code may belithographically printed on the sensor cover glass, or may be printed ona substrate and bonded to the sensor cover glass such as a 5 micronthick photographic film, or resist, substrate commercially available bycompanies such as Canyon Materials or Jenoptik.

FIG. 50 is an embodiment of the 2×2 and 3×3 systems using a singlesensor as a detector element for all channels. Only a side view is shownin FIG. 50. The detector 5016 and 5056 are for example very large APS(digital SLR) style sensors, or large SMpix to 14MPix commerciallyavailable CMOS image sensors. These sensors are on the order of 20 mm orlager in size (height and width dimensions), so 3×3 or 2×2 channels withdiameters of nominally 6 mm/channel will easily fit across a singlelarge format sensor. Baffling to control stray light is included in thissystem as 5018 and 5058 and is inherent in the lens holders or waferscale elements for each of the imaging channels. In configuration 5001each channel is provided a linear polarizer 5010, 5012, 5014 at 22.5degree angles to each other to form a polarization compass as disclosedpreviously. In configuration 5050 each channel is imaged through acontinuous substrate with grayscale intensity codes 5052. The grayscalecodes vary beneath each imaging channel as described earlier in FIGS. 42and 43 for sampling the two dimensional frequency space. Configuration5001 is a polarization compass and configuration 5050 is a codedlocalization system for performing sub-pixel location estimates.

1. Coded localization system, comprising: plurality of optical channelsarranged to cooperatively capture information from at least one objectonto a plurality of detectors; the optical channels having alocalization code to optically modify electromagnetic energy passingtherethrough, such that output data from the detectors are processableto determine localization of said object onto said detectors, and suchthat location of the object is determined more accurately than bydetector geometry alone.
 2. The system of claim 1, the localization codevarying as a function of spatial or range location of the data or theobject, and consisting of one of: intensity screen, wavelength, phasescreen, mask, aperture pattern, and polarization sensitive material. 3.The system of claim 1, the localization code comprising intensity codestructure occupying frequency spaces corresponding to object frequencyspectrum and having periodicity greater or equal to periodicity of thedetectors.
 4. (canceled)
 5. (canceled)
 6. (canceled)
 7. The system ofclaim 1, wherein at least one of the channels has no localization codeand measures average value of electromagnetic energy.
 8. The system ofclaim 1, further comprising a processing subsystem having a plurality ofalgorithms configured to process said data and provide at least one of arange estimate for the object and localization information for theobject.
 9. (canceled)
 10. (canceled)
 11. (canceled)
 12. (canceled) 13.The system of claim 1, further comprising: plurality of second opticalchannels arranged to cooperatively capture information from at leastsaid object onto a second plurality of detectors; the second opticalchannels having a localization code to optically modify electromagneticenergy passing therethrough, such that output second data from thesecond detectors are processable to determine localization of saidobject onto said detectors, and such that location of the object isdetermined more accurately than by second detector geometry alone,wherein the first and second data is processable to determine at leastone of angle and range to said object.
 14. (canceled)
 15. Codedlocalization polarization measurement system, comprising: plurality ofoptical channels arranged to cooperatively capture information frompartially polarized objects onto a plurality of detectors; the opticalchannels having a polarization code to uniquely polarize electromagneticenergy passing therethrough, such that output data from the detectorsare processable, to determine polarization pattern and determine if saidpattern differs from a model.
 16. The polarization measurement system ofclaim 15, the partial polarized data comprising the sky, the output datasampling polarization pattern of the sky.
 17. The polarizationmeasurement system of claim 15, further comprising a processingsubsystem having a plurality of algorithms and configured to processsaid data and provide compass heading.
 18. (canceled)
 19. Thepolarization measurement system of claim 17, the processing subsystemcomprising at least one of Fourier processing and an inner product witha signal of interest.
 20. (canceled)
 21. The polarization measurementsystem of claim 17, the processing subsystem adjusting exposure tomaximize partial polarization SNR.
 22. (canceled)
 23. The polarizationmeasurement system of claim 17, wherein the compass heading is usable in(a) a vehicle as feedback to stabilize heading, (b) a machine thataffects local magnetic field to determine heading.
 24. (canceled) 25.Coded localization navigation system, comprising: plurality of opticalchannels arranged to cooperatively capture information from a movingscene onto a plurality of detectors; each of the optical channelsdetermining its two dimensional change in motion of the scene; a rigidbody model coupling output from the channels to constrain a globalchange in location and orientation to a physical motion; and aprocessing subsystem for decomposing data from each channel into aglobal motion vector.
 26. The system of claim 25, the optical channelsproviding non-parallel measurements in at least one of pointingdirection, angle, orientation, magnification, and time.
 27. (canceled)28. (canceled)
 29. (canceled)
 30. The system of claim 25, the motionvectors usable in a vehicle for navigation.
 31. Coded localizationsystem, comprising: plurality of coded localization channels wherein thesystem has Fisher Information greater than an optical system withoutlocalization codes.
 32. A method of localizing optical data, comprising:cooperatively imaging at least one object onto a plurality of detectorswhile implementing localization codes on each optical channel; andprocessing output data from the detectors to determine localization ofsaid object onto said detectors such that location of the object isdetermined more accurately than by detector geometry alone. 33.(canceled)
 34. (canceled)
 35. A method for controlling a vehicle,comprising: utilizing a coded localization system to determine a presentheading of the vehicle; utilizing a global positioning system (GPS) todetermine a current location of the vehicle; determining a desiredheading for the vehicle by comparing the current location of the vehiclewith a desired destination; and adjusting the heading of the vehiclefrom the present heading to the desired heading.
 36. The method of claim35, the step of utilizing the coded localization system includingutilizing the coded localization system to identify a position of aknown object in reference to the vehicle.
 37. (canceled)